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SUMMARY 


Problem 


Actual  equipment  trainers  are  customarily  employed  in  training 
to  provide  competent  maintenance  technicians  to  the  field.  As  a  train¬ 
ing  device,  actual  equipment  does  not  readily  permit  the  controlled 
presentation  of  malfunctions  representative  of  the  trouble-shooting 
problems  which  occur  in  operational  settings.  Furthermore,  Air  Force 
policy  limits  the  type  of  faults  which  can  be  introduced  into  actual 
equipment  even  for  training  purposes.  Hence,  hands-on  experience 
most  often  provided  on  actual  equipment  trainers  is  generally  confined 
to  routine  procedural  tasks  rather  than  trouble-shooting  training. 

Less  expensive  real-time  simulators  do  possess  the  capability 
for  trouble-shooting  training,  incorporating  "hands-on"  practice  to 
increase  trouble-shooting  skill  on  samples  of  field-related  maintenance 
problems.  In  addition  to  improved  skills  training,  the  use  of  computer- 
based  training  simulators  has  the  potential  to  release  more  expensive 
actual  equipment  for  field  operations.  Simulators  in  comparison  to 
actual  equipment  provide  the  opportunity  to  deliver  instruction  which 
permits  greater: 

«  Actual  Task  Training 

•  Trouble-shooting  Practice 

•  Task  Specific  Remediation 

•  Learner  Diagnostics 

•  Learner  Motivation 

•  Objective  Evaluation 

•  Instructor  Control  (Task  &  Learner) 

•  Cost-Effective  Training 

Training  simulators  have  been  in  use  for  years.  Despite  this 
fact,  methodologically  sound  comparative  studies  of  the  training  and 
cost-effectiveness  of  simulators  and  actual  equipment  used  for  training 
are  conspicuously  rare.  Uncontrolled  factors  and  inadequate  evaluation 
methods  have  generally  produced  inconclusive  results.  Therefore,  the 
usual  finding  of  no  skills  and/or  life  cycle  cost  differences  as  a 
function  of  simulator  vs  actual  equipment  training  is  still  widespread. 
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Objective 


The  major  objective  of  the  present  evaluation  was  to  determine 
the  relative  effectiveness  of  a  3-dimensional  simulator  and  the  6883 
actual  equipment  test  station  on  dimensions  of: 

•  Instructional  Effectiveness/Efficiency 

•  Life  Cycle  Costs 

•  Attitudinal  Acceptance 

•  Field  Performance 

The  6883  test  station  was  chosen  for  the  simulation  project 
because  it  was  considered  representative  of  a  class  of  test  stations 
used  in  intermediate  level  training.  The  6883  simulator  was  expected 
to  provide  more  consistent  training  on  procedural  maintenance  and 
trouble-shooting  tasks  in  a  safe  training  environment  where  the 
probability  of  personal  Injury  and  costly  equipment  damage  are 
minimized. 


Approach 


The  approach  taken  was  to  compare  the  simulator  and  actual 
equipment  in  three  general  areas:  classroom  performance,  field  per¬ 
formance,  and  cost.  An  experimental  (simulator- trained)  and  control 
group  (actual-equipment  trained)  comparison  was  provided  within  a 
limited  practical  exercise  segment  (3  days)  of  the  F-lll  Avionics 
Maintenance  Course  at  Lowry  AFB. 

One  hundred  and  fifteen  F-lll  Avionics  Maintenance  trainees 
were  studied  during  the  evaluation  period.  Eighty-five  of  these 
students  were  Interviewed  in  field  assignments  subsequent  to  training. 

A  trouble-shooting  performance  test  was  developed  and  administered 
on  the  training  equipment  to  compare  the  training  adequacy  of  the 
simulator  to  the  actual  test  station  equipment.  Effects  of  training 
equipment  on  field  performance  were  assessed  by  a  "Projected  Job 
Proficiency  Test"  which  was  administered  upon  completion  of  the 
Instructional  training  block  and  follow-up  student  and  supervisor 
questionnaires  administered  in  the  field. 

A  life  cycle  cost  model  was  developed  and  implemented  to 
compare  the  cost  of  using  actual  and  simulated  training  equipment 
within  6883  instruction.  In  addition,  cost  scenarios  were  developed 
to  initially  describe  alternative  benefits  and  costs  associated  with 
three  simulator  training  roles:  (a)  generic  simulation  employed  solely 
throughout  an  entire  course,  (b)  replacement  of  ten  actual  equipment  test 
stations  with  up  to  ten  test  station  specific  simulators  and  (c)  sup¬ 
plemental  simulators  to  increase  skills  on  those  tasks  which  simulators 


are  shown  to  be  as  or  more  cost-effective  than  actual  equipment.  Addi¬ 
tionally,  course  materials,  STSs,  and  personnel  flies  were  reviewed  to 
determine  the  constancy  of  training  objectives  and  student  competency. 
All  equipment  malfunctions  were  recorded  and  their  Impact  on  training 
and  the  cost  model  was  assessed. 


Results 

The  simulator  and  the  actual  test  station  equipment  were  found 
to  be  equally  capable  of  training  students.  It  is  hypothesized  that 
a  number  of  environmental  factors,  which  were  noted,  may  have  reduced 
the  likelihood  of  observing  significant  preformance  differences  among 
experimental  groups.  These  factors  included  shifting  training  objec¬ 
tives  and  classroom  formats,  equipment  reliability,  changing  job 
requirements,  and  lack  of  a  clearly  defined  role  for  the  simulator 
in  training.  Specifically,  no  significant  differences  in  trouble¬ 
shooting  ability  or  field  performance  were  found  as  a  function  of 
training  or  testing  modes.  Time  to  complete  the  trouble-shooting 
performance  test  was  not  significantly  different  among  groups  and 
the  minor  differences  noted  were  attributed  to  the  slower  response 
time  of  the  simulator. 

Personal  interviews  and  subsequent  field  follow-up  question¬ 
naires  indicated  that  students  were  equally  comfortable  operating  either 
trainer.  Field-related  data  did  not  indicate  significant  differences 
in  field  performance,  based  on  training  mode,  regardless  of  field 
equipment  assignment. 

The  cost  comparison  Indicated  that  actual  equipment  costs  were 
approximately  twice  those  of  the  6883  simulator.  Since  the  6883  test 
station  was  one  of  the  most  reliable  stations,  it  was  hypothesized 
that,  from  a  total  course  systems  standpoint,  cost  savings  might  be 
significantly  higher.  Though  training  systems-r elated  cost  data 
were  not  available  at  this  time,  the  cost  of  alternative  role  strat¬ 
egies  for  Integrating  simulators  within  the  training  environment  was 
explored. 


Conclusions 

Students  trained  on  the  5883  3-D  simulator  performed  as  well  as 
students  trained  on  actual  equipment;  the  actual  training  benefits  of 
the  simulator  were  probably  not  realized,  however,  because  the  simulator 
was  designed  in  view  of  a  "replacement"  philosophy.  Given  this  design 
philosophy,  the  outcome  of  the  cost  comparison  between  trainers  be¬ 
comes  the  major  factor  in  future  procurement  decisions  given  approx¬ 
imately  equivalent  training  capability.  Given  a  "supplemental"  objec¬ 
tive,  Improved  performance  becomes  the  major  factor  considered  and  an 
analysis  of  cost  benefit  in  light  of  different  levels  of  performance 
skills  (i.e.,  training  effectiveness)  is  more  appropriate. 


While  other  training  alternatives  may  be  possible,  the  true 
potential  of  simulation  in  training  can  be  determined  only  by  a 
focused  research  effort.  A  new  training  strategy  which  maximizes 
the  training  potential  of  a  simulator  must  be  implemented  when  simu¬ 
lator  hardware  is  incorporated  into  an  existing  training  system. 


PREFACE 


This  second  Interim  Report  completes  Phase  II  of  the  cost  and 
training  effectiveness  evaluation  of  the  6883  simulator,  as  compared 
to  the  actual  6883  test  station  equipment,  for  training  intermediate 
level  F- 111  avionics  maintenance  personnel  at  Lowry  AFB.  The  plan 
for  this  evaluation  was  discussed  in  detail  in  the  1979  Phase  I 
Interim  Report  (TR-79-13).  The  final  report,  to  be  distributed  at 
the  conclusion  of  Phase  III,  will  include  evaluation  of  the  flat 
panel  training  simualtor  as  compared  to  the  3-dimensional  training 
simulator  and  the  actual  equipment  trainer.  It  will  also  include 
a  discussion  of  any  revisions  made  in  the  evaluation  plan  or  data 
collection  instruments. 

The  project  is  being  conducted  for  the  Air  Force  Human  Resources 
Laboratory,  Air  Force  Systems  Command,  United  States  Air  Force,  Brooks 
Air  Force  Base,  Texas.  This  evaluation  is  being  conducted  under  the 
technical  supervision  of  Dr.  Gerard  M.  Deignan,  Air  Force  Human  Resources 
Laboratory,  Project  Scientist  for  Program  2361-02-01.  Lt  Col  Downing 
is  the  Simulation  Program  Manager. 

The  evaluation  outlined  in  this  report  was  developed  by  the  Social 
Systems  Research  and  Evaluation  Division  of  the  Denver  Research  Institute, 
University  of  Denver,  Denver,  Colorado,  under  Contract  Number  F33615-78-C- 
0018.  Dr.  Louis  F.  Cicchinelli  is  the  Principal  Investigator  and  overall 
Project  Director. 
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CHAPTER  1 


INTRODUCTION 


Miller  (1974)  and  Miller  and  Gardner  (1975)  provide  an  extensive 
analysis  of  the  need  for  simulated  training  in  general  and,  specifically, 
for  the  development  of  the  6883  three-dimensional  (3-D)  simulator  em¬ 
ployed  in  this  study.  Factors  influencing  the  decision  to  proceed  with 
the  development  and  implementation  of  the  6883  simulator  included  the 
high  cost  of  actual  equipment  trainers  (AET) ,  their  low  reliability, 
safety  factors,  high  noise  levels  in  the  work  area,  and  the  limited 
scope  of  training  that  can  be  accomplished  on  AETs  because  appropriate 
malfunctions  cannot  be  inserted  without  extensive  and  costly  modifica¬ 
tions.  The  decision  to  simulate  the  6883  (Converter-Flight  Control 
Systems)  test  station  was  based  on  its  representativeness  of  a  class  of 
test  stations,  moderate  complexity  of  displays,  training  use  established, 
and  opportunity  for  comparative  evaluation. 


Previous  Research 


The  application  of  simulation  techniques  and  equipment  to 
maintenance  training  is  not  a  new  concept,  since  such  research  efforts 
date  back  to  the  early  1950s.  Yet  only  during  the  past  few  years  have 
extensive  efforts  been  made  to  realize  the  potential  of  the  approach. 
These  efforts  have  established  two  important  points:  maintenance  train¬ 
ing  simulators  can  be  cost  effective  (Miller,  1978;  Fink,  Shriver, 
Downing,  &  Miller,  1978;  Montemerlo,  1977)  and  can  provide  training 
comparable  to  the  actual  equipment  (Miller,  1974;  Hurlock  &  Slough, 

1976;  Crawford  &  Hurlock,  1976;  Wright  &  Campbell,  1975;  Daniels,  Datta, 
Gardner,  &  Modrick,  1975).  Furthermore,  for  tasks  requiring  problem 
solving  skills  and  cognitive  strategies,  simulation  may  be  the  best 
instructional  approach  (Kearsley,  1977). 

Despite  this  demonstrated  effectiveness,  there  is  a  growing 
belief  that  simulator  trainers  are  not  being  utilized  to  their  full 
capacity  and  that  much  of  the  simulator  training  research  is  subject 
to  methodological  problems.  Caro  (1977)  specifically  identifies  a 
number  of  factors  which  he  found  to  affect  Air  Force  simulator  training 
effectiveness.  Brock  (1977),  however,  found  that  the  introduction  of 
simulation  into  maintenance  training  could  reduce  the  amount  of  theory 
which  had  to  be  taught  to  obtain  a  particular  level  of  learning.  He 
noted  that  only  a  subset  of  required  skills  are  being  taught  in  existing 
maintenance  training  programs  and  that  the  addition  of  simulation 
methodology  may  only  change  the  subset  which  is  being  trained.  Miller 
(1978)  agrees  that  simulation  in  maintenance  training  may  result  in 
performance  capabilities  for  maintenance  program  graduates  which  are 


not  achievable  with  actual  equipment  trainers.  He  adds,  however,  that 
training  instructors,  or  those  who  are  responsible  for  acquisition  of 
training  systems,  do  not  seem  ready  to  acknowledge  this  potential.  Thus, 
the  key  to  simulation  efficacy  may  be  the  manner  in  which  it  is  utilized 
rather  than  the  choice  of  hardware. 

Defining  the  role  of  simulation  in  training  entails  a  compre¬ 
hensive  review  of  the  training  environment — its  objectives,  methods, 
goals,  and  means  of  assessment.  This  process  has  been  studied  exten¬ 
sively  by  the  Air  Force  under  the  designation.  Instructional  System 
Development  (ISD) .  To  date  ISD  has  been  utilized  very  little  in  the 
development  of  training  and  evaluation  instruments  for  simulator  pro¬ 
grams  . 

Generally,  attempts  have  been  made  to  determine  training  objec¬ 
tives  through  the  more  limited  task-analytic  approach.  In  theory,  such 
an  analysis  produces  training  which  is  more  job-relevant  and  makes  the 
use  of  simulation  more  practical.  However,  its  success  is  dependent 
on  an  accurate  determination  of  skill  level  requirements,  and  previous 
efforts  in  this  area  have  been  disappointing  (Brock,  1977;  Montemerlo, 
1976) .  One  reason  for  this  lack  of  success  may  be  the  failure  to 
include  student  trouble-shooting  abilities  as  a  task  objective. 

While  only  a  few  simulator  maintenance  training  programs  have 
been  in  operation  long  enough  to  assess  implementation  effects,  addi¬ 
tional  problems  become  apparent  when  the  ISD  approach  is  taken.  First, 
there  is  the  need  to  assess  the  fidelity  of  the  training  device  in 
order  to  establish  the  equivalency  of  training  contexts.  Otherwise, 
the  evaluation  design  may  incorporate  different  training  and  testing 
contexts  and  render  the  results  inconclusive  (Kearsley,  1977).  Special 
attention  must  be  given  to  psychological  fidelity  since  the  level  of 
realism  can  have  important  impacts  on  training  effectiveness  (Miller, 
1978).  Second,  there  is  the  need  for  a  criterion  measure  of  training 
effectiveness.  Previous  researchers  have  had  difficulty  in  establish¬ 
ing  such  a  measure  without  incurring  high  costs  (Miller,  1978).  And 
finally,  problems  can  arise  as  a  result  of  instructor  opposition. 
Instructors  may  see  their  teaching  role  threatened  by  the  incorporation 
of  simulation  methods  or  may  disagree  with  required  changes  in  the 
academic  structure. 


Since  1975,  the  ISD  process  has  been  specifically  prescribed 
for  use  in  all  new  instructional  systems  (AF  Manual  50-2,  1975).  But, 
while  several  ISD  schemes  have  been  developed  (Gagne,  1970;  Merrill, 
1971;  Markle  &  Tieman,  1972),  no  consensus  on  their  applicability  to 
simulator-based  maintenance  training  has  been  reached.  Without  this 
consensus  investigators  can  only  take  Miller's  (1978)  advice  to 
"concentrate  on  where  and  when  simulation  technology  can  best  be 
applied  and  under  what  conditions. " 


12 


The  present  study  necessarily  involved  some  of  the  issues  dis¬ 
cussed  above.  Concerning  the  use  of  simulation  in  maintenance  training, 
specifically,  it  was  important  to  consider  the  role  of  the  simulated 
training  device  in  training,  its  fidelity  to  AET,  and  user  acceptance 
of  the  new  equipment.  Furthermore,  the  cost  analysis  of  the  simulator 
and  AET  station  forced  examination  of  concepts  and  assumptions  that 
link  training  effectiveness  with  the  respective  system  costs.  Even 
though  the  3-dimensional  simulator  represented  only  one  of  a  family  of 
simulators  that  might  comprise  a  complete  training  equipment  system, 
the  cost  comparison  between  subsystem  components  provided  a  unique 
opportunity  to  isolate  unanticipated  requirements  (costs)  and/or  cost 
savings  that  might  accrue  at  the  system  level.  Thus,  the  overall 
objective  of  the  cost  analysis  was  to  contribute  to  the  evolution  of 
Air  Force  policy  regarding  simulators  and  the  use  of  simulation  in 
intermediate-level  maintenance  training. 


The  Role  of  the  6883  3-Dimensional  Simulator 

A  simulated  trainer  can  be  utilized  in  one  of  three  ways:  (a)  to 
provide  initial  contact  with  the  equipment  for  the  purpose  of  identi¬ 
fying  components;  (b)  to  complement  the  AET,  allowing  more  versatility 
of  training  while  lacking  the  complexity  of  the  AET;  or  (c)  to  replace 
the  AET  in  a  training  program,  delaying  technician  contact  with  the  AET 
until  the  period  of  on-the-job  training  (OJT) . 

Fink  and  Shriver  (1978)  identified  four  stages  of  learning: 

(a)  the  novice  stage,  (b)  the  uncoordinated  skills  stage,  (c)  the  co¬ 
ordinated  skill  in  a  training  environment  stage,  and  (d)  the  job  pro¬ 
ficiency  on  operational  equipment  stage.  They  suggest  that  the  simu¬ 
lated  training  device  should  play  a  role  during  the  third  stage,  the 
conclusion  of  which  usually  means  the  student  has  met  the  course 
requirements.  Use  of  the  AET  would  be  reserved  for  OJT,  or  stage  four. 
At  stage  three,  the  simulated  trainer  would  replace  the  AET  and  would 
allow  students  to  perform  operational  checks,  remove  and  replace 
components,  and  locate  malfunctions.  In  general,  the  simulator  would 
teach  the  logical  skills  required  to  isolate  malfunctions.  The  empha¬ 
sis  in  this  stage  is  on  teaching  the  "conceptual  aspects  of  trouble¬ 
shooting"  and  preparing  students  to  "train  or  work  on  real  equipment." 
Fink  and  Shriver  consider  this  to  be  the  most  cost-effective  way  to 
use  simulated  training  devices.  This  approach  to  using  simulators  in 
training  is  consistent  with  the  earlier  observations  of  Shriver  and 
Foley  (1974).  At  that  time,  thev  viewed  the  maintenance  activity  as 
having  a  very  strong  "organizing  factor."  A  "failure  of  the  performer 
to  correctly  organize  unrelated  discriminations  and  actions  will  result 
in  his  being  unable  to  perform  the  required  maintenance  activity." 

This  use  of  simulated  equipment  as  a  replacement  for  AET  is 
further  supported  by  Wheaton  et  al.  (1976)  and  by  Pieper  (1968,  1969), 
who  reported  that  "significant  additional  course  cost  reductions  were 


also  achieved  through  the  substitution  of  simulators  for  expensive  and 
scarce  tactically  configured  equipment."  Miller  and  Gardner  (1975) 
refer  to  the  6883  simulator  as  an  alternative  to  the  current  use  of 
AET. 


Fidelity  of  the  3-Dimensional  Simulator 

Equipment  (or  physical)  fidelity  of  simulated  trainers  is 
defined  as  the  comprehensiveness  and  level  of  detail  with  which  the 
real  world  (AET)  is  physically  represented  (Narva,  1978);  psychological 
(or  functional)  fidelity  is  defined  as  the  degree  to  which  the  trainee 
perceives  the  simulated  device  as  a  duplicate  of  the  actual  equipment 
(Miller  &  Gardner,  1975).  Psychological  fidelity  is  a  very  important 
consideration  in  evaluating  simulated  training  devices.  Wheaton  et  al. 
(1976)  proposed  that  the  importance  of  psychological  fidelity  was 
affirmed  by  observed  performance  differences.  They  hypothesized  that 
these  differences  were  due  to  the  fact  that  physically  identical  con¬ 
trols,  which  operated  differently  on  a  simulated  device  than  on  the 
AET,  had  a  detrimental  impact  on  transfer  of  learning. 

Fink  and  Shriver  (1978)  maintain  that  only  task-specific  com¬ 
ponents  of  training  equipment  should  be  simulated  and  they  should  have 
a  high  degree  of  psychological  fidelity.  Displays  and  operational 
controls  beyond  those  required  for  selected  maintenance  tasks  are  "at 
least  irrelevant  and  may  even  be  distracting  to  the  novice  technician." 
It  is  Narva's  (1978)  assumption  that  "the  potential  for  transfer  of 
training  will  increase  as  a  function  of  the  degree  to  which  the  required 
activities,  mental  and  physical,  are  represented  in  the  device  and  the 
degree  to  which  the  training  device  follows  'good'  practice  in  these 
activities."  Matheny  (1974)  maintains  that  psychological  equivalency, 
not  physical  equivalency,  results  in  positive  skills  transfer  and 
should  be  a  primary  factor  in  equipment  design  decisions.  The  design 
of  the  3-dimensional  simulator  involved  producing  an  instrument  that 
"looked  like  the  real  equipment  and  exhibited  psychologically  similar 
outputs  but  did  not  require  the  complex  internal  circuitry  of  the  real 
equipment"  (Miller  &  Gardner,  1975) . 


User  Acceptability  and  Training  Needs 

Fink  and  Shriver  (1978)  distributed  questionnaires  to  mainte¬ 
nance  instructors.  Their  findings  most  germane  to  a  discussion  of 
simulated  training  devices  were 

•  Most  instructors  were  willing  to  use  low-cost/fidelity 
training  devices  as  supplemental  to  AETs,  but  not  as 
replacements. 

•  Most  instructors  reported  a  heavy  reliance  on  AETs  and 
preferred  to  keep  it  that  way. 


•  Forty-three  percent  of  all  instructors  considered  the 
training  equipment  they  were  currently  using  to  be 
reliable.  This  figure  was  lower  for  electronic  main¬ 
tenance  instructors. 

•  Sixty-six  percent  of  the  instructors  used  the  AET  to 
teach  nomenclature  and  73  percent  used  it  to  teach 
parts  location. 

•  Sixty-five  percent  believed  that  students  need  to 
handle  actual  equipment  because  graphic  or  photo¬ 
graphic  representations  are  not  sufficient. 

•  Forty-two  percent  felt  trouble-shooting  should  be 
taught  on  the  AET,  while  53  percent  would  use  a 
trouble-shooting  logic  trainer,  but  only  if  that 
training  was  followed  by  training  on  the  AET. 

•  Fourteen  percent  of  those  responding  considered  the 
low  cost  and  easy  maintenance  of  simulated  training 
devices  as  primary  issues  in  any  decision  to  use 
simulated  training  devices. 

In  short,  it  was  found  that  instructors  prefer  to  use  proven 
instructional  devices  with  which  they  have  some  familiarity.  They 
would  accept  new  training  devices,  but  only  after  considerable  evidence 
had  been  collected  regarding  effectiveness  of  the  devices.  Since  the 
trainee  will  eventually  operate  an  actual  test  station  in  the  field 
assignment,  the  instructors  viewed  the  AET  as  a  superior  training  device 
despite  its  drawbacks  regarding  flexibility,  reliability,  safety,  and 
noise.  In  view  of  these  instructor  attitudes,  Fink  and  Shriver  (1978) 
expressed  reservations  about  expanding  the  role  of  instructors  and 
field  personnel  beyond  that  of  identifying  training  problem  areas. 

Pieper  (1968,  1969)  discussed  the  problem  of  determining  actual 
job  requirements  and  criticized  the  two  methods  most  often  used.  He 
suggested  that  one  method,  reviewing  instructional  materials,  assumes 
that  the  Plan  of  Instruction  (POI)  contains  all  field-related  tasks. 

This  is  not  a  valid  assumption.  He  also  pointed  out  that  surveying 
field  personnel,  the  second  method,  may  be  biased  by  senior  NCO  or 
commander  preferences  which  are  not  necessarily  the  most  effective 
methods  of  training.  The  task  analysis  necessary  for  the  design  of  the 
6883  3-dimensional  simulator  was  based  on  a  combination  of  these 
methods. 


CHAPTER  2 


EVALUATION  DESIGN 


This  chapter  provides  a  brief  outline  of  the  major  components 
of  the  evaluation  design  and  the  research  objectives.  This  information 
in  necessary  to  place  the  discussion  of  the  evaluation  environment 
presented  in  Chapter  3  in  a  proper  context. 


Research  Objectives 

The  objective  of  this  study  was  to  design  and  implement  a 
comprehensive  cost-training  effectiveness  evaluation  of  a  6883  3- 
dimensional  simulator,  as  compared  to  the  use  of  operational  6883 
test  station  equipment  for  training  intermediate  level  (I-level) 

F-lll  avionics  maintenance  personnel. 

The  evaluation  plan  outlined  in  this  section  was  developed  in 
view  of  information  collected  by  the  Denver  Research  Institute  (DRI) 
evaluation  team  over  the  initial  six  months  of  the  project.  A  com¬ 
plete  discussion  of  the  overall  design  employed  and  its  underlying 
rationale  was  presented  in  the  first  interim  report  for  this  project 
(Cicchinelli,  1979).  While  the  general  framework  remained  consistent 
with  the  proposed  contract  statement  of  work,  some  minor  modifications 
were  made  in  the  manner  of  implementation  based  on  an  analysis  of  the 
training  and  field  environments.  The  overall  design  was  divided  into 
three  components  to  facilitate  reference  to  the  three  major  evaluation 
components  originally  outlined  in  the  proposal.  Specifically,  the 
three  major  components  of  the  evaluation  were  classroom  performance, 
field  performance,  and  cost  analysis. 


Classroom  Performance 


The  basic  research  design  used  to  assess  classroom  performance 
as  a  function  of  training  mode  (actual  or  simulated  test  station  equip¬ 
ment)  is  shown  in  Figure  1.  Four  experimental  groups  were  defined  by 
the  two  training  modes  and  two  performance  testing  modes. 

TRAINING  MODE 
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Figure  1.  Research  Design  for  Comparing  Classroom  Performance 


Clearly,  performance  differences  in  operating  and  maintaining  actual 
test  stations  as  a  function  of  the  training  equipment  was  of  primary 
interest.  However,  two  testing  modes,  actual  and  simulator  equipment, 
were  used  because  it  was  expected  that  the  simulator  might  provide 
training  or  testing  capabilities  which  were  not  available  on  the  actual 
equipment.  Further,  it  was  necessary  to  determine  the  extent  to  which 
any  observed  differences  in  performance  were  due  to  familiarity  with 
the  test  equipment. 


Job  Performance 

The  design  shown  in  Figure  2  was  used  to  assess  the  impact  of 
training  mode  on  job  performance  in  the  field.  The  design  includes 
eight  experimental  groups  and  is  sensitive  to  the  possibility  that 
testing  classroom  performance  itself  constitutes  additional  training. 
Thus,  the  assessment  of  field  performance  must  be  conducted  in  view  of 
the  four  levels  of  training  resulting  from  the  various  combinations  of 
classroom  training  and  testing  modes. 

TRAINING 

Simulator  Actual  Equipment 

Simulator 

TESTING 

Actual 

Equipment 

FIELD 
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Figure  2.  Research  Design  Used  for  Comparing 
Field  Performance 


Cost  Analysis 

The  comparison  of  costs  associated  with  using  the  6883  actual 
test  station  and  the  6883  3-dimensional  simulator  for  training  was 
based  on  the  model  shown  in  Figure  3.  The  basic  model  consists  of  a 
matrix  of  six  major  cost  categories  and  two  components  of  life  cycle 
costs.  The  model  is  simply  the  "ingredients  approach"  discussed  by 
Levin  (1975)  in  which  cost  elements  are  identified  and  evaluated  con¬ 
sistent  with  the  ATC  acquisition  and  training  environment.  The  cost 
elements  or  ingredients  associated  with  each  cost  category  are  evalu¬ 
ated  either  as  one-time  costs  (primarily  Investment  Costs)  or  as 
Recurring  Annual  Costs,  consistent  with  AFHRL's  perspective  on  economic 
analysis  (Williams,  1977). 
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Life  Cycle 
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Figure  3.  Cost  Comparison  Model 


The  model  used  to  establish  the  life  cycle  cost  comparison  also 
provides  the  framework  for  evaluating  alternative  simulator  implementa¬ 
tion  strategies,  particularly  those  involving  proposed  changes  in  such 
"ingredients"  as  student  flow,  course  length,  and  time  to  complete  spe¬ 
cific  instructional  blocks.  In  this  sense,  the  model  is  general. 
Moreover,  it  will  also  be  used  to  establish  comparative  life  cycle 
costs  of  the  flat  panel  test  station  simulator  scheduled  for  evaluation 
during  1980  (Phase  III  of  this  project). 

To  assess  cost-effectiveness,  it  was  assumed  that  both  trainers 
have  equal  training  effectiveness.  Hence,  the  life  cycle  cost  compar¬ 
ison  between  trainers  will  indicate  which  is  the  most  cost-effective, 
i.e.,  the  trainer  exhibiting  the  least  total  cost  of  ownership.  This 
is  a  useful  approach  since  it  establishes  baseline  cost  data.  It  is 
also  consistent  with  the  original  simulator  design  objective  of  devel¬ 
oping  a  functional  (training)  replacement  for  the  6883  test  station 
(Miller  &  Gardner,  1975) .  The  discussion  of  the  cost  analysis  findings 
will  address  the  validity  of  the  equal  effectiveness  assumption,  however 
in  light  of  factors  (e.g.,  equipment  availability,  STS  standards)  found 
to  influence  the  training  effectiveness  of  the  respective  trainers. 

Additionally,  three  scenarios  are  presented  as  part  of  the  dis¬ 
cussion  of  the  cost  analysis.  They  outline  fundamentally  different 


^Line  items  associated  with  each  major  cost  category  are  pre¬ 
sented  in  the  "Results  and  Discussion"  section  of  the  report. 
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simulator  implementation  strategies  from  that  encountered  in  the  actual 
training  environment.  The  scenarios  are:  (a)  the  use  of  a  generic  test 
station  simulator,  (b)  the  use  of  combined  test  station/simulator  com¬ 
plements,  and  (c)  the  replacement  of  actual  equipment  trainers  with 
different  simulators.  These  scenarios  or  training  strategies  were 
chosen  because  they  are  thought  to  engage  the  relevant  Air  Force  sim¬ 
ulator  training  policy  options. 


Hypotheses  to  be  Tested 

This  evaluation  plan  was  designed  to  address  the  following 
hypotheses: 

•  Practical  training  on  the  6883  simulator  and  the  6883  test 
station  results  in  identical  performance  on  the  standard 
Air  Training  Command  block  tests  for  subsequent  training. 

•  Airmen  trained  on  the  6883  simulator  and  the  6883  test 
station  are  equally  accurate  in  solving  trouble-shooting 
problems. 

•  Airmen  trained  on  the  6883  simulator  and  the  6883  test 
station  are  equally  efficient  in  solving  trouble-shooting 
problems. 

•  Airmen  trained  on  the  6883  simulator  and  the  6883  test 
station  operate  the  actual  test  station  with  equal  pro¬ 
ficiency. 

•  Airmen  trained  on  the  6883  simulator  and  the  6883  test 
station  are  equally  familiar  and  comfortable  in  operating 
the  actual  test  station  without  supervision. 

•  Airmen  trained  on  the  6883  simulator  and  the  6883  test 
station  will  acquire  equivalent  job-related  experience. 

•  Airmen  trained  on  the  6883  simulator  and  the  6883  test 
station  will  be  equally  capable  of  operating  the  6883  test 
station  in  the  field. 

•  Airmen  trained  on  the  6883  simulator  and  the  6883  test 
station  are  equally  capable  of  operating  assigned  test 
stations,  other  than  the  6883  station,  in  the  field. 


CHAPTER  3 


THE  EVALUATION  ENVIRONMENT 


During  the  first  six  months  of  the  project,  the  DRI  evaluation 
team  conducted  an  analysis  of  the  activities  associated  with  the  F— 111 
avionics  maintenance  course  and  subsequent  field  assignments  for  grad¬ 
uates  of  the  course.  This  assessment  included  the  collection  and  review 
of  course-related  documents,  interviews  with  instructors  and  students, 
and  direct  observation  of  the  theory  and  practical  6883  classroom  pro¬ 
ceedings,  as  well  as  other  test  station  training  blocks.  The  review  of 
the  classroom  and  field  environments  uncovered  a  number  of  factors 
which  had  a  direct  impact  on  the  implementation  of  the  planned  evaluation. 
The  most  important  of  these  factors  are 

•  Training  objectives 

•  Format  of  training  program 

•  Availability  of  cost  data 

•  Reliability  of  training  equipment 

•  Status  of  the  6883  3-dimensional  simulator 

•  Assignment  and  OJT  in  the  field 

The  impact  of  each  of  these  factors  on  the  evaluation  is  discussed 
in  the  following  pages. 


Training  Objec tives 

Training  on  the  6883  Converter-Flight  Control  Systems  Test 
Station  occurs  as  part  of  a  23-week  intermediate  level  F— 111  avionics 
maintenance  course.  The  objectives  of  this  course  and  the  associated 
Specialty  Training  Standards  (STS)  have  been  in  a  continuous  state  of 
change  over  the  past  few  years. 

A  number  of  factors  have  contributed  to  the  need  to  modify 
course  objectives  and  content.  Perhaps  the  most  important  factor  is  the 
evolution  of  the  F— 111  aircraft  itself.  As  more  sophisticated  F— 111 
models  have  been  developed,  course  content  has  been  modified  to  include 
instruction  in  the  operation  and  maintenance  of  new  test  stations  capa¬ 
ble  of  testing  the  new  aircraft  systems.  Prior  to  1978,  students  were 
trained  as  test  station  operators  (course  ABR326X1B;  nine  days  of  6883 
theory  and  practical  training)  or  as  test  station  maintenance  personnel 
(course  ABR326X0B;  five  days  of  6883  theory  and  practical  training). 

In  1978,  these  career  options  were  integrated  into  a  single  career  path, 
and  a  new  Specialty  Training  Standard  (STS)  was  developed  to  reflect 


course  modifications.  In  this  combined  course  (Interim  Course  ABR326X1D) , 
theory  and  practical  training  on  the  6883  test  station  was  reduced  to  a 
total  of  eight  days.  More  recently,  still  another  plan  of  instruction 
was  developed  in  accordance  with  STS  ABR326X4A  which  became  effective  in 
April  1979.  This  course  includes  expanded  required  skill  levels  for 
3-level  training. 

Given  that  the  specifications  for  the  6883  3-dimensional  sim¬ 
ulator  were  developed  when  operation  and  maintenance  training  were  al¬ 
ternative  career  choices,  and  given  the  recent  merger  of  the  two  courses, 
we  felt  that  it  was  important  to  review  course  materials  used  over  at 
least  the  last  two  years  rather  than  the  proposed  one-year  period.  The 
approach  used  to  analyze  course  documentation  and  the  use  of  the  6883 
actual  test  station  in  training  was  to  compare  training  objectives  at 
three  discrete  points  in  time.  Specifically,  the  comparison  was  among 
instructional  objectives  used  when  maintenance  and  operation  of  the 
6883  test  station  were  separate  courses,  instructional  objectives  of 
the  1978  combined  interim  course  (ABR326X1F) ,  and  the  training  objec¬ 
tives  used  for  course  ABR326X4A,  which  was  in  place  during  this  evalua¬ 
tion  effort.  Briefly  stated,  our  analysis  of  training  objectives 
revealed  that  while  many  objectives  remained  the  same  over  time,  some 
objectives  (e.g.,  training  on  the  Yaw  Computer  LRU)  which  were  used 
to  define  the  simulator  capabilities  were  eliminated  due  to  the  signi¬ 
ficant  reduction  in  training  time  available.  Since  these  training 
exercises  were  no  longer  employed,  it  was  not  appropriate  to  include 
the  associated  simulator  capabilities  in  any  tests  of  student  per¬ 
formance. 

It  is  interesting  to  note  that  while  course  objectives  did 
change  over  time,  the  relevant  STS  requirements  did  not  substantially 
change.  This  finding  led  us  to  the  observation  that  the  specialty 
standards  are  general  enough  to  allow  for  interpretation,  depending  on 
one's  perspective.  That  is,  training  and  field  personnel  could  easily 
assume  that  somewhat  different  skills  are  associated  with  specific 
requirements,  such  as  "trouble-shooting."  Clearly,  the  lack  of 
specific  criteria  for  adequate  performance  poses  potential  problems 
for  an  evaluation  that  attempts  to  assess  training  effectiveness.  The 
problem  was  circumvented  in  this  study  by  considering  only  comparative 
training  effectiveness,  and  ignoring  the  more  basic  consideration  of 
training  adequacy.  In  short,  then,  this  study  answered  the  question, 

"How  do  simulator  trained  students  perform  as  compared  to  actual  equip¬ 
ment  trained  students?"  Although  an  assessment  of  the  overall  adequacy 
of  that  training  was  considered  beyond  the  scope  of  this  project,  we  do 
recommend  that  "adequacy  of  training"  be  a  primary  consideration  in  the 
development  of  Air  Force  training  policy. 


Format  of  Training  Program 


The  comparative  analysis  of  course  content  changes  over  the 
past  two  years  indicated  that  most  of  the  objectives  of  the  former 


maintenance  and  operations  courses  have  been  retained,  although  the 
time  allocated  to  each  has  been  greatly  reduced.  In  fact,  at  the 
beginning  of  the  current  performance  data  collection  phase,  only  two 
days  of  practical  training  on  the  6883  test  station  were  included  in 
the  F— 111  avionics  maintenance  course.  This  very  limited  contact  with 
the  6883  test  station  equipment  clearly  reduced  the  likelihood  that 
performance  differences  as  a  function  of  training  equipment  would  be 
observed.  Due  to  the  short  training  period,  it  became  necessary  to 
track  numerous  variables  which  might  obscure  actual  performance 
differences.  Much  effort  was  expended  in  controlling  for  individual 
differences,  minor  training  deviations,  instructor  and  supervisor 
differences,  etc.,  in  an  attempt  to  reduce  confounding  of  the  perform¬ 
ance  measures  by  changing  contextual  factors. 

DRI  originally  proposed  to  rely  heavily  on  existing  test  instru¬ 
ments  to  collect  relevant  data.  Further,  it  was  proposed  to  emphasize 
training  on  the  6883  3-dimensional  simulator  since  previous  test 
scores  from  students  trained  on  the  actual  test  station  equipment  would 
be  made  available  to  serve  as  baseline  data.  However,  after  completing 
the  review  of  courses,  it  was  found  that  the  merger  of  3ARB326X1D 
(operations)  and  3ARB326X0B  (maintenance)  was  not  the  only  major  change 
in  instructional  format  over  the  past  years.  In  fact,  these  two  courses 
have  undergone  numerous  modifications  and  the  new  combined  course  was 
continually  altered  even  during  the  data  collection  period.  Associated 
with  these  course  changes  were  modifications  in  the  test  instruments 
used.  In  1979,  the  6883  and  6886  test  station  practical  blocks  were 
merged.  These  formal  modifications,  together  with  numerous  training 
"deviations"  applied  to  nearly  every  class,  made  it  inappropriate  to  use 
test  scores  for  previous  classes  as  baseline  data. 

It  should  also  be  noted  that  for  some  instruction  blocks, 
previous  test  scores  did  not  exist.  For  example,  while  the  maintenance 
course  included  an  evaluation  of  student  performance  during  the  6883 
practical  block  based  on  a  performance  criterion  checklist,  no  similar 
testing  procedure  was  employed  in  the  operations  course.  Student 
performance  in  this  course  was  rated  as  "satisfactory"  or  "unsatis¬ 
factory"  on  the  basis  of  instructor's  observations  throughout  the  four 
days  of  training.  In  short,  there  were  no  standard  Air  Force  test 
instruments  available  which  could  be  used  to  compare  the  performance  of 
a  baseline  group  of  trainees  with  those  trained  on  the  6883  3-dimen¬ 
sional  simulator. 

Prior  to  the  data  collection  period,  practical  training  on  the 
6883  test  station  was  considered  a  single  block  of  instruction. 

However,  in  an  effort  to  reduce  the  overall  length  of  the  course,  it 
was  decided  to  merge  practical  training  on  the  6883  and  6886  test  sta¬ 
tions  into  the.  same  block  of  instruction.  To  maintain  the  same  amount 
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of  student-equipment  contact  in  one-half  the  time,  classes  were 
divided  into  two  groups.  One  group  trained  on  the  6883  test  station 
for  two  days  while  the  other  group  trained  on  the  6886  test  station. 
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By  reversing  the  groups  for  the  remaining  two  days  of  practical  train¬ 
ing,  all  students  completed  their  practical  training  on  both  test  sta¬ 
tions  in  four  days  rather  than  in  eight  days.  The  use  of  smaller  groups 
and  less  time  was  expected  to  result  in  the  same  level  of  training  as 
formerly  given.  Since  all  of  the  performance  testing  took  place  on  the 
day  after  the  practical  block  of  instruction  was  complete,  it  was  im¬ 
portant  to  control  for  the  sequence  of  training  received  by  each  stu¬ 
dent.  This  procedure  was  necessary  to  determine  if  6886  test  station 
training  between  6883  training  and  testing  had  a  detrimental  effect  on 
observed  performance. 

Two  additional  changes  on  the  F-lll  avionics  maintenance  train¬ 
ing  course  occurred  during  the  data  collection  period.  First,  the 
blocks  of  instruction  within  course  326X4A  were  rearranged.  Theory 
training  on  the  Electronic  System  Test  Station  and  the  Converter  Flight 
Control  Systems  Test  Station,  and  the  combined  practical  instruction  on 
these  two  test  stations,  were  positioned  earlier  in  the  sequence  of  in¬ 
struction  blocks.  This  modification  in  the  course  format  was  expected 
to  result  in  less  experienced  students  being  trained  in  the  6883  block 
of  instruction.  However,  the  change  did  afford  an  opportunity  to  ex¬ 
amine  more  carefully  the  relative  impact  of  actual  and  simulator  train¬ 
ing  on  subsequent  blocks  of  instruction.  The  data  collection  plan  was 
modified  to  carefully  note  any  deviations  in  the  sequence  of  training 
blocks.  Thus,  it  was  possible  to  examine  the  performance  of  students 
in  any  instruction  block  in  view  of  specific  prior  training  experiences. 

Second,  the  practical  block  of  instruction  on  the  Electronic 
System  and  Converter  Flight  Control  Systems  Test  Stations  was  expanded 
from  four  to  six  days:  three  days  of  practical  experience  on  each  test 
station.  The  extended  training  period  was  beneficial  in  that  it  in¬ 
creased  the  likelihood  that  any  existing  performance  differences  re¬ 
sulting  from  actual  versus  simulator  training  would  be  observed.  How¬ 
ever,  since  testing  occurred  after  the  entire  block  of  practical 
instruction  was  completed,  all  classes  necessarily  had  a  weekend  in¬ 
tervening  between  training  and  testing.  Since  this  delay  could  dif¬ 
ferentially  impact  observed  performance,  the  data  collection  plan  was 
modified  to  record  the  specific  nature  of  such  uncontrolled  time  lapses. 


Availability  of  Cost  Data 

The  essential  question  explored  in  the  Cost  Analysis  effort 
concerns  the  costs  of  ownership  cf  alternative  training  devices.  As 
might  be  expected,  some  difficulty  in  establishing  costs  of  ownership 
was  related  to  data  access  and  reliability.  Disaggregated  estimates 
of  the  development  costs  of  the  simulator  hardware,  software,  and 
courseware  available  from  the  manufacturer  are  not  releasable.  Thus, 
for  the  purposes  of  this  report,  the  acquisition  costs  of  the  simulator 
hardware,  software,  and  courseware  have  been  lumped  together  and 


treated  as  "sunk"  costs.  These  sunk  costs  have  been  included  in  the 
assessment  of  the  life  cycle  costs  of  the  respective  systems  as  Invest¬ 
ment  Costs  (rf.  Tables  15  and  16). 

The  contracted  cost  for  developing  and  manufacturing  the  6883 
simulator  was  $548,000.  The  contract  cost  itself  does  not  reflect 
the  total  expenditure  for  development  and  production  of  the  simulator 
because  corporate  funds  and  IR&D  were  used  in  years  prior  to  the  con¬ 
tract  award  to  develop  certain  aspects  of  the  technology  used.  An 
estimate  of  the  dollar  value  of  these  somewhat  indirect  costs  is  not 
available  from  the  manufacturer.  The  manufacturer  believes  that  the 
actual  costs  incurred  in  development  and  production  of  the  first  6883 
simulator  are  not  useful  as  an  estimate  of  the  costs  of  producing 
another  simulator.  The  6883  simulator  was  considered  to  be  a  research 
device  incorporating  features  inconsistent  with  preproduction  models 
and  current  computer  technology.  Our  interest  in  estimating  the  actual 
cost  of  the  6883  simulator  stemmed  from  a  concern  about  the  extent  to 
which  the  simulator  and  the  test  station  were  functionally  equivalent. 
For  example,  the  simulator  control  system  was  designed  to  be  expandable 
so  that  a  minimum  of  four  other  satellite  test  station  simulators 
comparable  to  the  6883  could  be  operated  simultaneously  from  the  same 
instructor  station.  Thus,  it  was  considered  important  to  be  able  to 
estimate  the  incremental  cost  of  hardware  and  software  development 
associated  with  the  additional  computer  that  provides  master  control 
for  simultaneous  operation  of  the  satellite  simulators. 

A  similar  difficulty  was  encountered  in  regard  to  estimating 
the  actual  test  station  costs.  ATC  maintains  two  Burroughs  D84  computer 
processors  and  related  peripherals  to  actually  "run"  automatic  fault 
isolation  tests  on  ten  test  stations  (including  the  6883)  used  in  F-lll 
training.  Developing  life  cycle  cost  information  on  these  Cenpac  com¬ 
puters  for  purposes  of  allocating  costs  to  the  operation  of  the  6883 
test  station  was  seen  to  be  a  project  equal  in  scope  to  the  cost  analy¬ 
sis  of  the  6883  test  station  and  the  effort  was  not  undertaken. 

These  two  problems  highlight  difficulties  associated  with 
attempting  a  cost  analysis  of  subsystem  components  in  the  absence  of 
a  total  system  analysis.  Although  the  6883  test  station  and  simulator 
are  discrete  elements  in  the  training  system,  it  is  not  known  how 
representative  they  are  of  other  system  components.  To  illustrate  this 
point,  consider  that  the  cost  of  maintenance  for  the  6883  test  sta¬ 
tion  was  based  on  maintenance  records.  For  purposes  of  comparison 
with  the  simulator,  we  have  no  way  of  judging  if  these  costs  are  repre¬ 
sentative  of  the  other  test  stations.  Similar  problems  of  representa¬ 
tiveness  arise  with  respect  to  the  costs  of  courseware  development. 

To  summarize,  then,  the  cost  analysis  should  be  understood  as  an  effort 
to  document  the  respective  cost  experiences  for  alternative  training 
devices  and  that  it  omits  important  considerations  and  detail  needed 
for  generalizations  about  the  cost  of  simulation  at  the  training  system 


level.  Hence,  a  fixed  effects  analysis  of  two  specific  training  devices 
is  provided  here. 


Reliability  of  the  Training  Equipment 

A  fundamental  issue  involved  in  the  evaluation  of  simulated 
training  devices,  and  the  6883  3-dimensional  simulator  in  particular, 
is  training  equipment  reliability.  Clearly,  equipment  malfunctions 
have  potential  impact  on  the  assessment  of  both  performance  and  cost. 
Fink  and  Shriver  (1978)  conducted  a  survey  of  maintenance  instructors 
and  ascertained  that  43  percent  of  those  interviewed  considered  the 
actual  equipment  trainer  (AET)  they  were  using  to  be  unreliable.  Un¬ 
scheduled  maintenance  downtime  for  a  one-year  period  ranged  from  a  low 
of  four  hours  to  a  high  of  6,654  hours. 

While  malfunctions  of  both  the  6883  test  station  and  simulator 
were  observed,  actual  training  time  was  lost  due  to  the  availability 
of  the  two  training  instruments.  The  frequency  of  equipment  failures, 
however,  was  sufficient  to  cause  disruption  of  routine  training  and 
data  collection.  The  major  impact  of  malfunctions  on  the  evaluation 
design  was  on  the  random  assignment  of  trainees  to  experimental  groups. 
Flexibility  in  evaluation  design  was  essential  to  allow  for  unexpected 
changes  in  the  training  schedule  and  to  minimize  the  potential  loss  of 
performance  data. 

These  periodic  interruptions  or  alterations  in  training  still 
had  some  minor  effects  on  our  testing  schedule,  although  a  more  signif¬ 
icant  impact  resulted  when  the  6886  test  station,  the  "partner"  test 
station  in  the  Converter-Flight  Control  System's  block  of  instruction 
was  not  operational  and  remained  unavailable  for  training  from  late 
August  to  mid-October  1979.  During  this  time,  four  classes  (21  stu¬ 
dents)  did  not  receive  6886  practical  training.  Since  our  research 
design  included  assigning  students  to  groups  by  training  sequence 
(whether  they  received  6883  training  before  or  after  6886  training), 
it  was  not  possible  to  place  these  21  students  in  the  established 
sequence  categories.  This  third  type  of  training  format  was  eliminated 
as  a  confounding  factor  when  the  analysis  of  training  sequence  found 
it  to  be  unrelated  to  performance  (rf.  p.  43).  It  should  be  noted  that 
impacts  on  6883  training  and  performance  due  to  equipment  failures  in 
other  training  blocks  prior  to  the  Converter-Flight  Control  blocks 
were  anticipated,  but  uncontrolled  in  this  research  design.  For 
example,  in  some  instances,  the  first  experience  students  had  with  an 
operational  test  station  came  in  the  Converter-Flight  Control  block. 

In  large  part,  equipment  reliability  defined  the  format  of  the 
training  approach  used  on  the  actual  test  station  equipment.  Unlike 
the  approach  with  preprogrammed  malfunctions  on  the  simulator,  the 
acquisition  of  practical  experience  was  dependent  on  pre-existing  or 


unexpected  equipment  failures.  This  was  particularly  true  of  the 
trouble-shooting  aspects  of  training.  In  its  extreme  forms  (which  were 
observed  during  the  experimental  period),  this  dependency  resulted  in 
very  limited  training.  At  those  times  when  all  test  replaceable  units 
(TRUs)  and  line  replaceable  units  (LRUs)  operated  without  malfunction, 
it  was  not  possible  to  demonstrate  trouble-shooting  techniques.  At 
other  times  when  a  specific  TRU  failure  caused  the  test  station  to  be 
inoperative,  no  training  was  possible. 

Finally,  equipment  malfunctions  during  training  sometimes 
resulted  in  different  training  sequences  for  various  classes.  To  main¬ 
tain  student  flow,  ATC  maintains  a  policy  of  moving  on  to  another  phase 
of  training  and  returning  to  the  block  of  instruction  that  was  missed 
when  the  equipment  becomes  operational.  In  some  instances,  when  speci¬ 
fic  AETs  remain  inoperable  for  long  periods  of  time,  a  class  may  not 
receive  any  training  in  a  particular  block.  A  "training  deviation" 
is  filed  for  the  class  and  graduation  occurs  as  scheduled.  With 
respect  to  the  evaluation  effort,  these  random  variations  in  the 
training  sequence  made  it  impossible  to  assess  the  efforts  of  prior 
training  on  performance  in  the  6883  practical  block  of  instruction. 


Status  of  the  6883  3-Dimensional  Simulator 


The  6883  simulator  was  not  completely  operational  upon  arrival 
at  Lowry  in  June  1978.  Much  time  was  devoted  to  "debugging"  training 
exercises  and  modifying  the  hardware  to  improve  the  system  (Schneider, 
1978).  While  these  activities  did  improve  the  performance  of  the 
system,  the  formative  evaluation  also  delayed  the  start  of  this  data 
collection  effort  and  resulted  in  additional  costs  which  would  normally 
not  be  incurred  in  the  acquisition  of  a  production  model  of  the 
simulator.  These  costs  were  monitored  and  particular  attention  was 
given  to  them  when  implementing  the  cost  analysis  (rf.  p.  72 )•  All 
modifications  to  the  simulator  were  monitored  carefully  to  ensure  that 
both  the  capabilities  and  limitations  of  the  equipment  were  considered 
in  designing  student  performance  measurement  instruments. 


Assignment  and  OJT  in  the  Field 

Once  trainees  are  assigned  to  the  field,  they  enter  the  train¬ 
ing  phase  called  on-the-job  training  (OJT).  Field  training  is  rela¬ 
tively  informal  and  is  designed  to  develop  the  skills  needed  to  operate 
and  maintain  the  assigned  test  station. 

Field  site  visits  conducted  in  the  course  of  the  project, 
revealed  that  the  number  of  technicians  available  and  their  tech¬ 
nical  competency  upon  arrival  determined  the  extent  and  tvpe  of  OJT 
received.  Due  to  the  individualized  approach  to  providing  OJT  in  the 
field,  it  was  not  possible  to  use  the  extent  and  type  of  OJT  required 


as  an  indicator  of  training  effectiveness.  Another  method  of  assessing 
field  performance  considered  was  to  record  the  number  and  cost  of  re¬ 
placement  parts  requested  and  used  by  new  technicians  to  perform  test 
station  and  LRU  repairs.  Utilizing  this  indirect  measure  of  training 
effectiveness  was  not  feasible  either  due  to  the  complexities  of  ob¬ 
taining  such  data  in  the  field.  In  short,  it  was  difficult  to  isolate 
clear  measures  of  long-range  impact  of  simulator  training. 

It  was  anticipated  that  the  performance  of  simulator-trained 
personnel  and  actual  equipment-trained  personnel  would  be  compared 
at  specified  intervals  of  time  in  the  field.  The  effects  of  on-the- 
job  training  were  expected  to  be  present  and  constant  in  both  groups 
and,  therefore,  any  differences  in  performance  could  be  attributed  to 
the  mode  of  training.  Clearly,  with  variable  amounts  of  OJT,  this 
assumption  was  not  valid.  In  fact,  if  the  originally  proposed  time 
series  sampling  framework  was  used,  differences  in  performance  due  to 
training  would  be  reduced  as  time  in  the  field  increased.  Therefore, 
in  order  to  obtain  some  measure  of  the  long-range  impact  of  simulator 
training  on  performance,  it  was  necessary  to  devise  a  method  of  esti¬ 
mating  job  proficiency  prior  to  field  assignment,  and  to  collect  sub¬ 
jective  ratings  of  field  performance  from  supervisors  shortly  after 
the  field  placement  (within  about  two  weeks) . 

From  discussions  with  ATC  staff  and  field  personnel,  it 
became  apparent  that  field  assignments  are  made  in  view  of  the  momen¬ 
tary  demand  for  specific  automatic  test  station  operators.  Thus,  at 
best,  only  a  small,  undetermined  number  of  airmen  trained  to  operate 
the  6883  test  station  were  expected  to  be  assigned  to  the  6883  or  6873 
test  stations  in  the  field.  Although  the  nature  of  the  field  assign¬ 
ments  remained  a  design  variable,  it  was  clear  from  the  outset  that 
extremely  unequal  sample  sizes  for  students  assigned  to  6883  and 
other  test  stations  would  be  obtained. 


CHAPTER  A 


METHODOLOGY 


This  chapter  provides  a  discussion  of  all  data  collection 
instruments  developed  to  assess  both  classroom  and  field  performance 
as  well  as  costs.  The  data  collection  and  management  techniques 
employed  are  also  outlined.  In  addition,  this  chapter  includes  a  dis¬ 
cussion  of  some  preliminary  analyses  conducted  to  verify  the  research 
design  and  performance  test  administered.  An  analysis  of  all  data 
collected  is  provided  in  Chapter  5  of  this  report. 


Assignment  to  Treatment  Groups 

A  total  sample  of  115  F— 111  avionics  maintenance  trainees 
participated  in  this  study.  Students  were  assigned  to  the  eight  treat¬ 
ment  groups  defined  by  training  sequence  and  the  two  levels  of  training 
mode  and  test  mode.  Each  "training  class"  was  partitioned  into  two 
approximately  equal  groups  for  assignment  to  levels  of  training 
sequence  and  mode,  whereas  the  "student"  was  the  unit  of  assignment  to 
a  level  of  test  mode.  It  should  be  noted  that  although  training 
sequence  was  initially  considered  in  the  assignment  strategy,  subse¬ 
quent  analysis  indicated  that  performance  did  not  vary  as  a  function  of 
training  sequence  (rf.  p.  A5) .  The  distribution  of  students  among  the 
four  remaining  experimental  groups  is  shown  in  Figure  A. 
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Figure  A.  Distribution  of  Students  Among 
Experimental  Groups 


The  assignment  of  students  to  groups  was  essentially  random. 

The  schedule  was  adjusted  monthly  in  an  effort  to  maintain  approxi¬ 
mately  equal  group  sizes  throughout  the  data  collection  period.  Since 
the  adjustments  made  were  necessary  due  to  random  fluctuations  in  class 
size,  unscheduled  deviations  from  the  course  timetable,  and  unpredicted 
equipment  malfunctions,  the  overall  attempt  to  randomly  assign  students 
to  experimental  groups  was  not  compromised. 

Despite  this  effort  at  random  assignment,  it  was  conceivable 
that  observed  performance  would  be  a  function  of  student  aptitude. 


To  justify  attributing  any  observed  differences  to  training  mode,  it 
was  necessary  to  consider  pre-existing  individual  differences  among 
students.  To  examine  the  possibility  that  a  matched  sample  strategy 
should  be  included  in  the  assignment  plan,  the  relationship  between 
aptitude  scores  and  scores  on  prior  instruction  block  exams,  and 
performance  in  the  6883  theory  and  practical  blocks  of  instruction  was 
analyzed  for  a  sample  of  104  trainees  over  the  previous  two  years  . 

Table  1  shows  that  this  analysis  indicated  that  only  modest  correlations 
existed  between  aptitude,  as  measured  by  Air  Force  testing  protocols, 
and  the  6883  theory  training  block  scores.  No  relationship  between 
aptitude  and  practical  block  scores  was  found.  This  may  be  due  to  the 
"pass-fail"  system  used  to  rate  practical  performance.  It  is  important 
to  note  that  the  analysis  of  prior  performance  was  based  on  previous 
block  examinations  which  have  been  continually  revised  to  reflect 
changes  in  course  content.  Therefore,  these  findings  are  not 
necessarily  generalizable  to  the  test  instruments  used  in  this  evalua¬ 
tion  effort.  Although  no  systematic  bias  due  to  aptitude  was  expected 
to  confound  the  performance  measures  and  a  matched  sample  design  was 
not  used,  group  equivalence  was  again  examined  as  part  of  the  analysis 
(rf.  p.  45).  No  significant  relationship  between  aptitude  scores  and 
performance  was  found. 

The  final  issue  considered  which  relates  to  the  assignment  of 
students  to  treatment  groups  is  the  procedure  used  to  make  field 
assignments.  Evaluator  control  over  this  factor  was  not  possible  since 
assignments  are  made  on  the  basis  of  demand  in  the  field.  Thus,  data 
collected  subsequent  to  field  assignments  were  analyzed  separately  for 
trainees  assigned  to  the  6883  or  6873  test  station  and  trainees  assigned 
to  other  automatic  test  stations.  It  was  expected  that  the  group  sizes 
would  be  unequal  on  this  factor.  In  fact,  our  field  follow-up  data 
for  the  first  63  students  indicated  only  12  were  ever  assigned  to  the 
6883  or  6873  test  stations,  and  only  six  of  these  were  still  assigned  to 
the  6883  or  6873  stations  at  the  time  of  our  second  follow-up. 


Data  Collection 


Consistent  with  the  three  components  of  the  evaluation  design 
discussed  earlier  in  this  report,  three  general  types  of  information 
were  collected:  classroom-related  data,  field-related  data,  and  cost- 
related  data.  The  strategy  for  collecting  classroom  and  field  data 
was  carefully  designed  to  minimize,  as  much  as  possible,  the  number  of 
interruptions  of  normal  Air  Force  procedures. 

Classroom-Related  Data 


As  a  result  of  a  cooperative  agreement  between  ATC  and  HRL,  one 
day  at  the  end  of  the  Converter-Flight  Control  Systems  practical  block 
of  instruction  was  made  available  to  DR1  for  data  collection.  This 
temporary  departure  from  the  normal  training  schedule,  effective  for 
the  duration  of  this  project,  allowed  data  to  be  collected  without 
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TABLE  1 


Correlations  between  Student  Characteristics  and 
Prior  Performance  and  6883  Test  Station  Performance 


Factors 

6883  Theory  Block 

6883  Practical  Block 

Age 

.09 

.19 

Sex 

.  17 

.15 

ASVAB-General 

.26* 

.36 

ASVAB-Mechanical 

.11 

.10 

ASVAB-Administrative 

.09 

.22 

ASVAB-Electronics 

.29* 

.28 

AFQT 

.29* 

.37 

Fundamentals  of  Electronics 

.41** 

.36 

Avionics 

.41** 

.23 

CENPAC 

.  10 

.01 

DATAC 

.06 

.21 

CATE 

.51** 

.37 

Test  Station  6863 

.39** 

.33 

Test  Station  6886 

.41** 

.22 

*Significant  at  .05  level. 

**Significant  at  .01  level. 
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altering  the  usual  6883  training  protocol.  Allocating  one  day  for 
testing  immediately  following  the  6883  training  was  extremely  helpful 
to  this  evaluation  effort  for  a  number  of  reasons.  First,  the  logis¬ 
tics  of  integrating  the  data  collection  into  ongoing  training  were 
simplified.  Second,  the  use  of  the  actual  or  simulated  6883  test 
stations  occurred  as  it  normally  would  if  there  had  been  no  evaluation 
effort.  Third,  the  availability  of  testing  time  immediately  following 
6883  training  eliminated  the  possibility  that  observed  performance 
would  be  adversely  affected  by  intervening  training.  Finally,  collect¬ 
ing  performance  data  from  each  student  at  the  same  point  in  the  train¬ 
ing  sequence  ensured  that  all  students  had  similar  levels  of  training 
at  the  time  of  testing.  Clearly,  the  cooperation  of  ATC  in  this 
matter  was  essential  to  the  evaluation  effort  and  DRI  is  appreciative 
of  their  assistance. 

In  order  to  collect  all  information  required  from  an  entire 
class  of  students  (usually  six)  in  a  single  day,  it  was  necessarv 
to  administer  the  performance  tests  and  conduct  the  interviews  in  a 
different  order  for  each  student.  Administration  of  the  trouble-shoot¬ 
ing  performance  test  for  the  first  student  was  followed  by  the  personal 
interview.  During  the  time  that  any  one  student  was  being  administered 
the  trouble-shooting  performance  test,  the  remaining  students,  who  had 
either  completed  or  were  awaiting  their  turn  to  be  tested,  were  given 
a  Projected  Job  Proficiency  Test.  When  all  students  in  a  class  had 
completed  the  performance  test,  the  Projected  Job  Proficiency  Test, 

•,and  had  been  interviewed,  the  evaluation  day  was  complete. 

Prior  to  the  evaluation  day,  DRI  staff  reviewed  ATC  records  for 
the  current  class  to  determine  if  any  training  deviations  had  occurred 
or  if  any  unscheduled  maintenance  periods  were  recorded.  The  type  of 
training  received  was  also  recorded  and  it  was  confirmed  that  the 
experimental  group  assignments  were  still  valid. 

Since  the  F— 111  maintenance  course  was  restructured  (rf.  p.  21), 
it  was  necessary  for  ATC  to  revise  test  instruments  to  reflect  changes 
in  course  objectives.  Thus,  it  was  not  appropriate  to  compare  the 
performance  of  previous  classes  trained  on  the  6883  actual  test  station 
equipment  to  the  performance  of  classes  trained  on  the  6883  simulator. 

Our  evaluation  plan  considered  this  factor  and  a  design  element  which 
allowed  us  to  collect  6883  performance  data  from  students  trained  on 
the  actual  equipment  as  well  as  on  the  simulator  was  included. 
Additionally,  all  performance  scores  on  instructional  blocks  before 
and  after  the  6883  blocks  were  recorded  for  students  included  in  this 
study.  The  following  sections  provide  a  brief  discussion  of  the  major 
data  collection  instruments  developed  and  other  sources  of  classroom- 
related  data. 

Student  trouble-shooting  ability  test.  This  test  was  designed 
so  that  it  could  be  administered  on  either  the  actual  tost  station 
equipment  or  the  simulator.  The  primary  focus  of  this  test  was  on 
trouble-shooting  skills;  the  problem  selected  included  the  identification 


and  correction  of  a  malfunction  in  an  LRU  and  a  simultaneous  malfunction 
in  the  test  station.  The  test  was  designed  to  examine  the  ability  of 
a  student  to  perceive  the  test  station  and  its  associated  LRUs  as  an 
integrated  system.  The  two-step  nature  of  the  malfunction  was  chosen 
to  test  the  logical  process  of  trouble-shooting.  The  test  was  dis¬ 
similar  to  the  lessons  available  on  the  6883  3-dimensional  simulator  and 
to  the  majority  of  malfunctions  encountered  on  the  6883  actual  equip¬ 
ment.  Thus,  the  task  was  new  to  all  students. 

The  trouble-shooting  test  items,  which  were  individually  scored 
by  trained  Denver  Research  Institute  personnel,  are  shown  in  Section 
A-l  of  Appendix  A.  Each  item  constitutes  a  descrete  step  that  a  stu¬ 
dent  had  to  complete  in  order  to  identify  the  malfunction.  In  addition 
to  recording  the  completion  (and  noncompletion)  of  the  required  steps, 
the  nature  of  all  errors  committed  was  noted  on  the  scoring  form.  An 
analysis  of  these  errors  was  conducted  to  isolate  any  differences 
between  students  trained  on  the  6883  S-dimensional  simulator  and  actual 
test  station  equipment.  In  addition  to  the  measures  or  performance, 
time  to  complete  various  portions  of  the  test  and  degree  of  instructor 
assistance  required  during  testing  were  also  recorded.  It  should  be 
noted  that  at  specific  points  in  the  test  protocol,  instructor  prompt¬ 
ing  was  necessary;  the  amount  of  information  provided  was  controlled 
as  well  as  possible.  As  Shriver  and  Foley  (1974)  discussed,  however,  it 
is  difficult  to  ensure  that  consistent  information  is  provided  because 
few  technicians  use  the  same  logic  tree  to  solve  a  problem.  Information 
that  is  useful  to  one  person  might  be  unenlightening  to  others. 

The  emphasis  on  trouble-shooting  performance  testing  was  vali¬ 
dated  by  field  personnel  at  both  Plattsburgh  and  Cannon  AFBs .  Twenty- 
four  avionics  personnel,  having  at  least  two  years  of  field  experience, 
were  asked  to  rate  the  relevancy  of  the  6883-related  STS  standards  to  job 
performance.  Trouble-shooting  test  station  malfunctions  was  considered 
the  most  important  job  skill.  Respondents  also  rated  the  ability  to 
analyze  a  specific  problem  logically  as  a  critical  skill.  Understanding 
the  operation  of  the  test  station,  its  component  TRUs,  and  associated 
signal  flow  were  considered  necessary  to  effectively  trouble-shoot 
the  station  and  LRUs.  The  mean  ratings  of  all  STS  standards  by  field 
personnel  are  shown  in  Table  2. 

Student  and  instructor  interviews.  Each  student  participating 
in  the  study  was  interviewed  immediately  after  they  completed  the 
trouble-shooting  test  to  obtain  their  opinions  and  attitudes  concerning 
the  use  of  simulators  in  training.  The  interview  guide  is  shown  in 
Section  A-2  of  Appendix  A.  In  addition  to  the  collection  of  student 
data,  each  trainee's  personal  file  was  reviewed.  Demographic  informa¬ 
tion,  aptitude  scores,  and  prior  and  subsequent  block  scores  were 
recorded. 

The  ATC  Instructor  Questionnaire  was  administered  in  December 
1979  to  20  instructors  in  the  F— 111  training  program  at  Lowry  AFB. 

They  were  asked  to  respond  to  questions  regarding  the  general  use  of 
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TABLE  2 


Summary  of  STS  Ratings  by  field  Personnel 

STS  Item  Rating 

General  (n=24) 

Purpose  and  function  of  test  station  (27a)  3.7 

Theory  and  signal  flow  of  switching  complex  ( 2 7b 1 )  4.2 

Theory  and  signal  flow  of  switching  control  unit  (27b2)  3.85 

Theory  and  signal  flow  of  ratio  input  filter  ( 2 7b 3)  3.15 

Theory  and  signal  flow  of  FCS  adapter  (27b4)  3.05 

Theory  and  signal  flow  of  CDU  adapter  (27b5)  3. 1 

Theory  and  signal  flow  of  SCU  controller  (27b6)  3.45 

Theory  and  signal  flow  of  parallel  digital  adapter 

(2 7b 7)  3.35 

Theory  and  signal  flow  of  serial  digital  adapter  (27b8)  3.5 

Theory  and  signal  flow  of  digital  interface  unit  (27b9)  3.6 

Theory  and  signal  flow  of  signal  converter  simulator 

(27b 10)  3.55 

Perform  confidence  test  for  test  station  (27c)  2.) 

Perform  diagnostic  testing  of  the  test  station  ( 2 7 d  1 )  3.65 

Trouble-shoot  malfunctions  of  the  test  station  (27d2)  4.35 

Perform  maintenance  of  the  test  station  (27d3)  3.7 

Verify/calibrate  the  test  station  (27d4)  3.0 

Feel  and  Trim  ( n = 2 3 ) 

General  information  and  electrical  characteristics  (28a 1)  3.4 

Data  flow  and  interface  with  test  station  (28bl)  4.0 

Test  and  inspect  (28cl)  2.85 

Rating  scale:  1  2345 

irrelevant  slightly  relevant  very  critical 
relevant  relevant 
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TABLE  2  (cont.) 


STS  Item  Rating 

Isolate  malfunctions  (28dl)  4.05 

Repair  (28el)  3.4 

Multiplexer  Converter  (n=23) 

General  information  and  electrical  characteristics  (28a3)  3.65 
Data  flow  and  interface  with  the  test  station  (28b3)  4.0 

Test  and  inspect  (28c3)  2.85 

Isolate  malfunctions  (28d3)  4.05 

Repair  (28e3)  3.35 

Flight  Control  Computers  (FCC)  (n=23) 

General  information  and  electrical  characteristics  (28a4)  3.1 
Data  flow  and  interface  with  the  test  station  (28b4)  3.7 

Test  and  inspect  (28c4)  2.75 

Isolate  malfunctions  (28d4)  3.2 

Repair  (28e4)  3.1 


Other  LRUs  (please  specify  any  other  LRU  or  Astro  Electronics)  (n=2) 
General  information  and  electrical  characteristics  (28a5)  3.5 


Data  flow  and  interface  with  the  test  station  (28b5)  3.0 
Test  and  inspect  (28c5)  2.0 
Isolate  malfunctions  (28d5)  2.5 
Repair  (28e5)  2.5 


TABLE  2  (cont.) 


STS  Item 


General  Capabilities 


Rating 


(n=23) 


Practice  housekeeping  consistent  with  safety  of  personnel 

and  equipment  (3b)  3.55 

Apply  safety  precautions  pertaining  to  using  tools  and 

equipment  (3a)  3.7 

Use  of  technical  manuals  as  a  source  of  information  for 

performing  maintenance  and  inspection  (4b)  4.1 

Selection,  care,  and  use  of  hand  tools  (10a)  3.2 

Apply  safety  precautions  pertaining  to  high  voltage  (NA)  4.3 

Use  of  maintenance  data  collection  forms  (7c)  2.65 


Additional  Tasks  Not  Listed 

j  Reading  of  test  data  charts  with  the  aid  of  shop 

systems  TO  5  (n=3) 

Reading  flight  line  malfunctions  from  350  tags  and 

associate  with  failure  while  on  test  station  4  (n=5) 

Understanding  of  TRU  and  LRU  circuitry  relationships 
and  approaching  trouble-shooting  tasks  in  a  logical 
manner  rather  than  using  a  shotgun  method  4.5  (n=5) 

Theory  and  signal  flow  of  phase  sensitive  converter  TRU  5  (n=2) 

Theory  and  signal  flow  of  ratio-transformer  TRU  5  (n=l) 

Theory  and  signal  flow  of  station  miscellaneous  switching 


simulators  in  training  and  about  the  6883  3-dimensional  simulator  in 
particular.  Due  to  schedule  constraints,  annual  leave,  and  educational 
commitments,  it  was  necessary  to  allow  the  instructors  to  complete  the 
questionnaires  when  possible  and  return  them  to  DRI  staff.  This  ques¬ 
tionnaire  is  provided  in  Section  A-3  of  Appendix  A. 

Course  content  and  equipment  malfunction.  The  effort  to  deter¬ 
mine  the  nature  of  actual  test  station  use  in  training  included  the 
collection  and  review  of  course-related  documents,  interviews  with 
instructors  and  students,  and  direct  observation  of  classroom 
proceedings . 

Additionally,  the  specific  training  received  by  students  on  both 
the  6883  actual  test  station  and  simulator  was  recorded.  The  training 
received  was  categorized  according  to  the  TRUs  of  the  AET  or  simulator 
addressed  in  the  lesson,  the  LRUs  which  were  included  in  the  training 
period,  and  the  complexity  of  the  training.  Training  equipment  malfunc¬ 
tions  for  both  pieces  of  equipment  were  also  recorded.  This  was  accom¬ 
plished  by  reviewing  maintenance  department  records  for  the  AET  and 
reviewing  the  daily  log,  maintained  by  ATC  and  HRL  personnel,  for  the 
simulator.  In  addition,  appropriate  instructors  were  interviewed  to 
corroborate  the  written  records  for  both  trainers.  An  emphasis  in 
compiling  equipment  malfunction  data  was  on  those  malfunctions  which 
in  any  way  interrupted  or  interfered  with  the  training  of  the  avionics 
technicians.  Therefore,  the  data  presented  does  not  include  all  equip¬ 
ment  downtime,  but  only  those  instances  or  periods  relevant  to  the 
study.  This  information  served  a  dual  purpose:  to  track  any  effects  of 
malfunctions  on  training  and  to  provide  information  for  cost  analysis. 

Data  was  also'  collected  regarding  deviations  in  training  for 
students  during  the  period  of  their  6883  training.  Deviations  are 
given  to  students  who,  for  any  of  several  reasons  including  equipment 
malfunctions,  have  not  received  all  of  the  required  training  time  in  a 
particular  block.  Reasons  for  deviations  range  from  adverse  weather 
conditions  to  Base  Commander  authorized  events  to  equipment  breakdowns. 

Field-Related  Data 


It  was  originally  planned  to  assess  the  possible  impact  of 
simulator  training  on  actual  job  performance.  As  previously  discussed 
(rf.  p.  27),  differences  in  job  performance  as  a  function  of  training 
equipment  would  be  obscured  by  the  variable  nature  of  OJT.  To  circum¬ 
vent  this  problem,  we  developed  a  paper-and-pencil  test  as  a  predictor 
of  future  job  performance.  This  instrument  was  administered  on  the 
test  day  immediately  following  6883  training. 

Subsequent  to  permanent  field  assignment,  all  technicians  who 
participated  in  this  evaluation  project  and  their  field  supervisors 
completed  follow-up  questionnaires.  Since  former  students  could  be 
assigned  to  any  one  of  three  work  shifts  or  were  on  temporary  leave 
for  medical  or  educational  reasons  locating  them  was  itself  a 


difficult  task.  Field  personnel  were  instrumental  in  this  aspect  of 
the  data  collection  effort.  They  assumed  the  responsibility  of  dis¬ 
tributing  the  questionnaires  and  forwarding  all  information  to  DRI. 
Generally,  the  technicians  and  their  supervisors  completed  the  ques¬ 
tionnaires  at  their  assigned  station.  Only  about  15  minutes  of  each 
person’s  time  was  required. 

Predicting  job  proficiency.  In  order  to  insure  the  validity 
of  the  Projected  Job  Proficiency  Test,  a  criteria  of  job  proficiency 
was  established.  Questions  were  solicited  from  supervisors  which  they 
felt  would  reflect  what  is  expected  of  newly  trained  personnel.  A 
total  of  75  questions  (and  answers)  were  received.  From  these  and 
questions  obtained  from  Lowry  AFB  instructors,  a  70-item  pencil-and- 
paper  test  of  job  proficiency  was  developed  (Section  A-4  of  Appendix 
A).  Since  time  constraints  did  not  permit  pretesting  the  instrument 
on  student  populations  prior  to  implementation,  the  following  pre¬ 
cautions  were  taken:  (1)  the  test  instrument  was  shown  to  ATC  instruc¬ 
tor  personnel  and  the  6883  curriculum  supervisor  for  comments  regarding 
the  applicability  and  validity  of  the  questions  utilized,  and  (2)  an 
item  analysis  capability  has  been  incorporated  into  the  data  collection 
instrument  to  permit  an  item  analysis  of  the  test  instrument.  Prior 
to  implementation,  the  test  instrument  required  only  one  modification, 
as  suggested  by  ATC  personnel. 

Field  technician  survey.  To  generate  field  performance  data, 
the  originally  proposed  field  performance  task  was  replaced  with  an 
interview  questionnaire.  Technicians  who  had  participated  in  the 
evaluation  program  and  were  subsequently  assigned  to  the  field  were 
asked  to  rate  the  adequacy  of  their  ATC  F-lll  training  and  its  rela¬ 
tionship  to  their  field  work.  Those  technicians  who  had  contact  with 
the  simulator  during  training  were  asked  specifically  to  rate  the  ade¬ 
quacy  of  simulator  training.  Section  A-5  of  Appendix  A  includes  this 
questionnaire. 

Field  supervisor  survey.  Technicians  in  OJT  are  assigned  a 
supervisor  who  assists  them  in  becoming  familiar  with  the  responsibili¬ 
ties  of  field  assignments  and  who  rates  their  performance  during  the 
OJT  period.  The  supervisor  may  work  at  the  assigned  test  station  with 
the  technician  or  may  only  be  available  to  answer  questions  on  an 
as-needed  basis.  Personnel  limitations  require  that  one  supervisor 
be  responsible  for  more  than  one  technician  in  most  cases.  These 
supervisors  were  asked  to  rate  the  performances  of  technicians  in 
areas  of  housekeeping,  use  of  testing  equipment,  knowledge  of  LRU  and 
TRU  circuit  flow  and  operation,  and  trouble-shooting  initiative.  There 
were  a  total  of  25  items  on  which  each  technician  was  rated. 

As  originally  planned,  former  students  were  to  be  rated  after 
three,  six,  and  nine  months  of  field  experience.  It  was  felt,  however, 
that  by  six,  certainly  by  nine  months,  any  effects  of  training  would 
have  been  diluted  by  formal  OJT  and  practical  experience  obtained  in 
the  field.  Thus,  field  follow-up  data  were  collected  in  September  and 
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October  1979  (after  2-20  weeks  of  field  experience)  during  on-site  in¬ 
terviews  by  DRI  staff.  In  January  1980  (after  2-34  weeks  of  field  ex¬ 
perience),  data  was  again  collected  on  all  students  using  a  mail  ques¬ 
tionnaire.  Treating  time  in  the  field  as  a  continuous  variable  allowed 
us  to  examine  performance  differences  as  a  function  of  time.  The 
supervisor  rating  form  is  included  as  Section  A-6  of  Appendix  A. 

Cost-Related  Data 

A  comparison  of  life  cycle  costs  associated  with  the  6883  simu¬ 
lator  and  AET— under  the  assumption  of  equal  training  effectiveness — 
was  conducted  to  determine  which  trainer  was  more  cost-effective.  Since 
the  6883  simulator  is  one  of  the  first  such  devices  introduced  into  the 
intermediate  level  maintenance  training  environment,  this  analysis  will 
also  serve  as  a  point  of  reference  for  a  discussion  of  simulation  options 
available  in  Air  Force  maintenance  training. 

The  question  of  which  trainer  is  more  cost-effective  can  be  com¬ 
plicated  when  different  training  equipment  utilization  patterns  for  the 
simulator  and  AET  also  produce  differences  in  training  effectiveness. 

For  example,  the  more  costly  equipment  alternative  might  still  be  the 
most  cost-effective  because  the  equipment,  together  with  its  utilization 
strategy,  produces  proportionately  greater  training  effectiveness. 

These  complications  were  not  encountered  in  the  evaluation  be¬ 
cause  the  total  training  environment  was  not  disturbed  by  the  introduc¬ 
tion  of  the  simulator.  That  is,  with  the  introduction  of  the  simulator, 
no  new  training  strategy  was  imposed  which  could  have  been  expected  to 
result  in  differences  in  training  effectiveness.  Thus,  the  comparison 
of  only  the  life  cycle  costs  of  the  two  equipment  items  is  a  realistic 
way  to  establish  which  of  the  trainers  is  more  cost-effective. 

2 

It  should  be  noted  that  the  term,  "cost-effectiveness,"  is  used 
here  in  a  limited  sense:  .  .  ."which  of  two  or  more  equivalent  training 
systems  has  the  least  total  cost  of  ownership."  The  notion  of  cost- 
effectiveness  of  simulation  in  training  is  a  broad  concept  that  presumes 
(a)  equal  effectiveness  of  both  simulators  and  actual  equipment  trainers 
and  (b)  that  student  performance  can  be  gauged  by  objective  performance 
criteria  associated  with  student  task  performance  on  the  actual  equip¬ 
ment.  In  this  context,  cost-effectiveness  analysis  of  simulation  train¬ 
ing  would  Include  a  determination  of  the  marginal  utility  of  the  simula¬ 
tor;  that  is,  at  what  point  does  greater  use  of  the  simulator  no  longer 
reduce  training  costs  for  the  simulator-AET  combination.  Orlansky  and 
String  (1977)  develop  this  important  topic  in  their  review  of  Cost-Ef¬ 
fectiveness  of  Flight  Simulators  for  Military  Training.  This  issue  is 
not  engaged  in  the  present  cost  analysis  because  the  amount  of  training 
could  not  be  systematically  varied  (for  institutional  reasons),  nor 
could  student  performance  of  trouble-shooting  tasks  be  ascertained  using 
existing  Air  Force  performance  criteria.  This  assessment  is  not  inten¬ 
ded  to  be  a  criticism,  rather  it  is  a  comment  on  the  maturity  of  simu¬ 
lation  development  for  the  maintenance  training  field. 


Life  cycle  cost  comparison.  The  approach  taken  to  the  develop¬ 
ment  of  the  life  cycle  cost  comparison  considers  that  the  simulator  is 
a  replacement  alternative  to  AET.  The  respective  cost  streams  for  each 
of  the  trainers  were  calculated  for  a  projected  15-year  life.  Since 
the  comparison  used  1978  as  the  reference  year,  AET  costs  incurred 
prior  to  1978  were  adjusted  according  to  the  annual  inflation  rate  to 
establish  investment  costs  in  1978  dollars.  Operating  costs  were  dis¬ 
counted  at  10  percent  annually  to  establish  net  present  value  in  1978 
dollars.3  Figure  5  displays  the  framework  for  cost  comparison  and  the 
major  categories  for  which  input  data  were  acquired. 


Cost  Categories 

Simulator 

AET 

Facilities 

Equipment 

Industrial  Materials 

Personnel 

Students 

Miscellaneous 

TOTAL  (construction  $) 

NPV  (1978) 

Cost  Effectiveness  ($ 
per  student-hour) 

Figure  5.  Life  Cycle  Cost  Comparison  Framework 


Related  cost  models.  Two  studies  concerned  witli  the  cost  of 
training  systems  have  been  particularly  useful  in  this  work.  The  Naval 
Training  Analysis  and  Evaluation  Group  (Braby,  Henry,  Parrish,  & 

Swope,  1975)  developed  a  cost  model  which  included  methods  for  evaluat¬ 
ing  the  elements  in  each  cost  category  shown  above.  A  computer  program 
for  calculating  total  training  cost  was  also  provided.  The  TAEC  model 
and  its  cost  element  estimation  procedures  would  have  been  used  in  this 
work  except  for  two  drawbacks:  (1)  the  model  contains  insufficient 
detail  concerning  the  equipment  acquisition  life  cycle  phase  that  was 
considered  important  here,  and  (2)  the  emphasis  given  the  TAEC  model 
was  one  of  predicting  costs  of  alternative  systems  for  the  purpose  of 
cost  minimization  and  not  for  predicting  total  costs  of  system 
ownership . 


3This 


discount 


rate  is  consistent  with  0MB  Guidelines 


(Circular 


A94) . 


A 


The  Air  Force  Human  Resources  Laboratory,  Wright  Patterson  AFB 
(Eggemeier  et  al.,  1979)  completed  a  life  cycle  cost  estimation  for 
F-16  simulated  and  AET  equipment  maintenance  training  systems  in  which 
they  used  the  Braby  model  as  a  starting  point.  In  addition,  they  incor¬ 
porated  a  logistics  support  cost  model  to  estimate  elements  in  the 
equipment  cost  category.  The  latter  model  was  originally  developed  by 
the  Simulator  System  Program  Office  (SPO)  to  estimate  costs  for  air¬ 
crew  simulators.  The  principal  value  of  the  logistics  support  model 
is  that  it  can  be  used  to  estimate  certain  cost  elements  based  on  func¬ 
tional  relationships  derived  from  historical  acquisition  data.  This 
technique  was  helpful  in  the  present  study  for  estimating  sustaining 
investment  costs  associated  with  the  6883  AET  operation.  In  addition, 
the  F-16  analysis  also  provided  guidance  in  estimating  elements  in 
other  cost  categories.  These  cost  elements  were  directly  related  to 
ongoing  efforts  since  F-16  training  will  be  conducted  at  Lowrv  AFB 
when  these  trainers  become  available. 

Thus,  the  cost  model  presented  in  this  study  is  perhaps  best 
thought  of  as  a  method  for  organizing  and  tabulating  cost  elements 
("ingredients")  for  which  historical  data  are  available.  The  sources 
of  the  historical  data  for  the  AET  station  used  in  this  cost  comparison 
included  the  3450th  TTG  Avionics  Branch  and  the  Logistics  Support  Group 
(Lowry  AFB).  The  sources  of  data  for  the  simulator  include  AFHRL/TT 
(technical  monitor  for  the  simulator  contract)  and  the  manufacturer  of 
the  simulator. 


Data  Management 

After  students  completed  their  F— 111  training  and  performance 
data  were  collected,  coding  forms  designed  by  DRI  were  completed  and 
the  data  were  entered  into  a  computer  tape  file  for  subsequent  analysis. 
At  this  point,  all  data  except  field  follow-up  data  had  been  collected 
for  each  student. 

The  data  collected  regarding  training  deviations,  training 
received,  unscheduled  maintenance,  and  personal  interviews  were  also 
organized  for  analysis  by  DRI.  When  the  evaluation  period  was  con¬ 
cluded,  the  impact  of  these  various  factors  on  training  and  performance 
was  assessed. 

Field  follow-up  data  were  collected  and  checked  against  a  list 
of  the  technicians,  their  base  assignments,  and  length  of  time  in  the 
field.  A  search  was  initiated  to  find  those  technicians  who  were  not 
at  the  base  to  which  they  were  assigned  at  graduation. 
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CHAPTER  5 


RESULTS  AND  DISCUSSION 


The  primary  focus  of  this  cost  and  training  effectiveness  eval¬ 
uation  was  to  examine  classroom  and  field  performance  as  a  function  of 
training  mode  (AET  or  simulator)  and  to  assess  the  costs  of  using  the 
alternative  types  of  training  equipment.  This  chapter  provides  a  dis¬ 
cussion  of  all  evaluation  findings  together  with  the  outcome  of  all 
supporting  analyses  undertaken. 


Preliminary  Analyses 

In  order  to  insure  that  any  observed  differences  in  student 
performance  could  be  attributed  to  the  training  equipment  used,  it 
was  necessary  to  consider  two  possible  sources  of  confounding:  train¬ 
ing  sequence  and  pre-existing  student  differences. 

Impact  of  Training  Sequence  on  Performance 

Variations  in  the  sequence  of  training  on  the  6883  and  6886 
test  stations  were  inherent  in  the  ATC  environment.  Although  testing 
followed  both  segments  of  training,  for  some  students  6886  training 
preceded  both  6883  training  and  subsequent  DRI  testing,  for  others, 
it  occurred  between  6883  training  and  the  testing.  To  control  for 
possible  6886  interference,  this  factor  was  therefore  added  to  the 
basic  experimental  design,  resulting  in  two  levels  each  of  training 
mode,  testing  mode,  and  training  sequence. 

Failure  of  the  6886  test  station  equipment  late  in  the  exper¬ 
iment  made  this  scheme  less  tenable,  for  21  of  the  115-student  sample 
did  not  receive  any  6886  training.  Two  solutions  to  this  problem  were 
considered.  One  was  to  add  a  third  level  of  the  training  sequence 
factor  and  call  it  "none."  It  was  decided,  however,  that  adding  a  third 
level  to  the  training  sequence  variable  would  result  in  a  loss  of  sta¬ 
tistical  power  without  contributing  to  the  analysis  of  interference 
effects.  The  more  viable  approach  taken  was  to  examine  the  impact  of 
training  sequence  independent  of  the  21  students  who  received  no 
training  on  the  6886  test  station.  If  there  were  no  apparent  differ¬ 
ences  in  student  performance  as  a  function  of  training  sequence  and  if 
all  three  types  of  sequence  were  equally  distributed  across  training/ 
testing  modes,  then  it  would  be  reasonable  to  collapse  across  the 
sequence  factor  for  all  remaining  analyses. 

Table  3  presents  descriptive  statistics  for  the  three  overall 
trouble-shooting  test  measures—total  test  score,  total  time  to  test 
completion,  and  a  rating  of  the  degree  of  assistance  required — as  a 
function  of  training  sequence  and  experimental  conditions.  Note  that 


Assistance  Required  Total  Time  Total  Test  Score 


TABLE  3 


there  is  a  discrepancy  between  the  reported  sample  size  of  89  and  the 
94  students  who  actually  participated  in  the  study.  This  difference 
resulted  from  the  fact  that  five  students  solved  the  experimental 
problem  prematurely — they  detected  the  blown  fuse  without  proceeding 
through  the  required  steps.  In  such  cases,  data  were  recorded  for  the 
particular  items  completed,  but  these  students  were  not  included  in 
analyses  of  overall  proficiency. 

Subsequent  two-way  analyses  of  variance  on  these  data  revealed 
no  significant  main  effects  of  sequence  F(1 ,81)< 1  for  both  total  score 
and  time  to  completion.  Nor  were  the  ratings  of  degree  of  assistance 
required  distributed  unequally  between  6886-first  and  6883-first 
training  sequences  (X^(2)  =  1.251,  r  =  .535).  The  lack  of  relationship 
between  training  sequence  and  overall  performance  was  even  more  appar¬ 
ent  when  point-biserial  correlations  were  computed.  The  coefficients 
relating  sequence  to  total  score,  total  time,  and  assistance  were  .015, 
-.059,  and  .116,  respectively.  It  seems  reasonable  to  conclude  that 
intervening  6886  training  had  no  appreciable  effect  on  experimental 
testing. 


The  second  important  consideration  was  whether  or  not  the  final 
three  sequence  levels  (6886-first,  6883-first,  and  none)  were  distri¬ 
buted  equally  across  the  four  experimental  conditions.  If  not,  it  might 
be  argued  that  the  presence  or  absence  of  6886  training  per  se  could 
influence  the  findings.  A  tabulation  of  the  actual  and  expected  fre¬ 
quencies  of  students  across  conditions  is  provided  in  Table  4.  Using 
a  chi  square  analysis,  no  evidence  was  found  to  suggest  that  levels  of 
sequence  were  distributed  unequally  across  the  training/testing  modes 
(X2(6)  =  5.533,  P  =  .477). 

In  summary  then,  the  potential  interference  of  intervening  6886 
training  upon  6883  performance  was  not  substantiated.  The  three  levels 
of  training  sequence  obtained  appear  unrelated  to  the  four  critical 
experimental  conditions.  Since  some  change  in  the  evaluation  design 
was  necessitated  by  the  6886  training  deviations  and  the  expansion  of 
the  sequence  factor  to  three  levels  would  add  little  while  reducing 
statistical  power,  training  sequence  was  ignored  as  a  factor  in  all 
subsequent  analyses. 

Tests  for  Bias  in  Croup  Assignments 

Having  eliminated  training  sequence  from  further  analyses,  it 
was  necessary  to  determine  if  any  meaningful  pre-existing  differences 
existed  between  students  assigned  to  the  four  remaining  experimental 
conditions.  The  biasing  factors  considered  were  sex,  aptitude,  and 
achievement  in  prior  training  blocks. 

BJas  due  to  sex.  Although  it  is  difficult  to  predict  just  how 
the  training  and  testing  modes  might  differentially  affect  men  and 
women,  sex  was  considered  an  important  factor  in  any  possible  biasing. 


raining  Sequence 


TABLE  4 


Distribution  of  Students  Across  Experimental  Conditions 
as  a  Function  of  Eventual  6886/6883  Training  Sequence. 


Mode  of  Training-Testing 

Actual/ 

Actual 

Actual/ 

Simulator 

Simulator/ 
Ac t  ua 1 

Simulator/ 

Simulator 

Total 

6886  then 

6883 

12(12) 

9(11) 

12(11) 

47 

6883  then 

6886 

12(12) 

10(12) 

11(11) 

13(11) 

47 

none 

3(5) 

7(5) 

8(5) 

3(5) 

21 

Total 

30 

29 

28 

28 

115 

Note:  Expected  frequencies  for  each  cell  are  in  parentheses. 
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Table  5  presents  the  distribution  of  students  across  experimental  groups 
by  sex.  Not  surprisingly,  a  chi  square  analysis  of  these  data  failed 
to  show  any  significant  relationship  between  sex  and  group  assignment 
(x^(3)  =  2.637,  p  =  .451).  Thus,  there  is  no  reason  to  believe  that 
males  and  females  were  unequally  distributed  among  experimental  groups. 

Bias  due  to  inherent  ability.  Perhaps  a  more  pertinent  question 
is  whether  or  not  these  groups  of  students  differed  in  their  basic 
abilities.  Two  sets  of  test  scores  were  used  to  address  somewhat  dif¬ 
ferent  aspects  of  this  question.  The  first  set  of  scores  included  the 
results  of  five  standard  Ai,r  Force  screening  instruments  used  to  measure 
student  aptitudes.  A  one-way  analysis  of  variance  was  completed  for 
each  type  of  score  by  assignment  level  and  the  resulting  values  are 
presented  in  Table  6. 

A  word  of  caution  must  be  offered  with  regard  to  interpreting 
the  exact  probability  levels  reported.  Despite  the  fact  that  these 
scores  cannot  be  considered  truly  independent  because  the  same  students 
produced  the  scores  for  each  measure,  some  adjustment  must  be  made  to 
avoid  capitalizing  on  chance.  Miller  (1966)  suggested  that  even  when 
such  multiple  tests  are  not  independent,  a  significance  level  of 
“/n  (where  «  =  some  probability  and  n  =  the  number  of  tests  performed) 
on  each  test  will  guarantee,  with  a  probability  of  at  least  1  -  a,  that 
no  true  null  hypothesis  will  be  rejected  (Fisher's  method  of  adjustment). 
For  the  present  situation,  then,  this  method  suggests  that  the  meaning¬ 
ful  significance  level  should  be  . 0 l ( . 05 / 5 ) . 

Admittedly,  this  approach  is  rather  conservative,  especially 
for  the  purposes  of  identifying  possible  biasing  factors.  If  Fisher's 
adjustment  is  not  used,  however,  less  weight  must  be  given  to  an 
occasional  significant  result  if  it  occurs  along  with  a  number  of 
results  that  are  nonsignificant.  Thus,  the  marginally  significant 
difference  (p  =  .09)  between  groups  on  AFQT  scores  is  of  dubious 
importance. 

Rather  than  dismiss  this  finding  out  of  hand,  however,  a  corre¬ 
lational  analysis  was  performed  to  determine  to  what  extent,  if  any, 
these  AFQT  scores  were  related  to  the  three  primary  experimental  per¬ 
formance  measures.  The  resulting  correlation  coefficients  were  .065 
for  total  test  score,  -.176  for  total  time  to  completion,  and  -.143 
for  the  degree  of  assistance  required.  These  correlations  are  not  sig¬ 
nificant  even  at  p  =  .1  (df  =  80).  Thus,  any  difference  in  aptitude, 
as  measured  by  the  AFQT,  is  not  related  to  subsequent  trouble-shooting 
performance . 

Bias  due  to  prior  achievement.  Even  more  directly  related  to 
the  issue  of  student  ability  and  group  assignment  was  the  second  set 
of  measures  which  was  comprised  of  the  11  prior  block  scores  obtained 
during  avionics  training.  Practical  test  scores  for  the  Electronics 
System  TS  and  Converter-Flight  Control  TS  instructional  blocks  were 
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TABLE  5 


Observed  and  Expected  Frequencies  of  Males  and  Females 
by  Experimental  Training/Testing  Modes 


Mode  of 

Tra in ing / Te  s  t ing 

Male 

Students 

"  ■  ■  1 

Female 

Students 

Total 

Actual/Actual 

27(26) 

3(4) 

30 

Actual/Simulator 

23(25) 

6(4) 

29 

Simulator/Actual 

26(27) 

2(4) 

28 

Simulator /Simulator 

24(24) 

4(4) 

28 

Total 

100 

15 

115 

Note:  Expected  frequencies  are  in  parentheses. 


TABLE  6 


Aptitude  Scores  as  a  Function  of  Experimental  Group 


omitted  from  further  study  since  all  students  received  a  grade  of 
"satisfactory."  The  remaining  nine  scores  were  included  in  an  analysis 
of  variance  (Table  7).  < 


The  argument  made  above  concerning  significance  levels  must 
again  be  considered  when  interpreting  these  findings.  Since  nine  tests 
were  made,  Fisher's  method  suggests  that  a  functional  alpha-level  of 
.006(.05/9)  be  used.  By  this  criterion,  no  significant  differences  in 
group  assignment  were  found  as  a  function  of  prior  achievement. 

However,  for  purposes  of  identifying  bias,  those  block  scores  with 
marginal  differences  warranted  further  investigation.  Specif ically , 
correlations  were  computed  between  the  experimental  dependent  measures 
and  the  Data  Logic  and  CATE  block  scores.  For  total  test  score,  the 
correlations  with  Data  Logic  and  CATE  scores  were  .143  and  .005, 
respectively;  for  total  time  to  completion  they  were  -.177  and  -.176; 
and  for  the  degree  of  assistance  required  they  were  -.017  and  -.157 
Since  these  correlations  are  not  significant  at  p  =  .1  (df  =  80),  there 
is  no  evidence  that  these  prior  achievement  measures  are  related  to 
performance  on  the  experimental  task. 

To  summarize,  three  factors  were  considered  as  sources  of 
possible  bias  in  the  assignment  of  students  to  the  four  training/test- 
ing  conditions  of  this  study:  sex,  aptitude,  and  prior  ATC  performance. 
The  groups  did  not  differ  significantly  on  any  of  these  measures. 
Further  correlational  analyses  showed  that  even  in  those  cases  where 
it  might  be  argued  that  a  slight  bias  existed,  variables  were  not  mean¬ 
ingfully  related  to  performance  on  the  trouble-shooting  test. 

Analysis  of  Trouble-shooting  Performance 

A  central  issue  to  be  addressed  by  this  investigation  was 
whether  simulator  and  actual  equipment  training  would  result  in  equal 
levels  of  student  performance  on  a  practical  trouble-shooting  problem 
as  might  be  encountered  in  the  field.  Because  of  stringent  selection 
criteria  used  for  entry  into  avionics  technician  training,  it  was  not 
surprising  to  find  that  students  were  fairly  homogeneous  with  respect 
to  their  prior  performance  in  the  program.  Neither  could  marked  dif¬ 
ferences  in  subsequent  performance  be  expected  after  such  a  brief 
training  manipulation.  Thus,  the  test  developed  to  isolate  any  train¬ 
ing  differences  had  to  be  particularly  sensitive  to  differences  in  the 
training  equipment  (rf.  pp.  26-27).  To  reiterate,  the  test  consisted 
of  a  timed,  serial,  hands-on  29-item  task  which  allowed  three  overall 
dependent  measures  of  performance:  a  total  score,  the  total  time 
necessary  for  test  completion,  and  a  three-point  rating  of  the  degree 
of  assistance  required  for  test  completion. 

Trouble-shooting  as  a  Function  of  Training/Testing  Modes 

Table  8  summarizes  performance  as  a  function  of  training  and 
testing  modes. 
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Overall  Trouble-Shooting  Test  Proficiency  as  a 
Function  of  Training  and  Testing  Modes 
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While  all  115  students  may  have  completed  any  particular  test 
item,  only  109  overall  test  scores  are  shown.  This  is  due  to  the 
method  of  scoring  the  test.  For  those  students  who  were  ,d  '  ,  let 
whatever  reasons,  to  isolate  the  cause  of  the  malfunction  l!  ■  :i  i  -wn 
fuse)  without  completing  all  of  the  intermediate  steps,  no  ov  tall 
measures  were  recorded.  Instead,  data  for  the  items  completed  were 
included  only  insofar  as  they  were  in  correct  sequential  order.  It  i- 
interesting  that  all  six  students  who  prematurely  detected  the  Mown 
fuse  were  in  the  actual  equipment  test  situation  where  the  fuse  hold. a 
was  necessarily  activated  prior  to  the  tost.  Also,  all  six  had  'u  en 
trained  on  the  actual  6883  test  station  and  this  fin. line  mav  nileet 
their  familiarity  with  actual  test  station  operation,  ex:n  r  ienct  v.-  it:: 
blown  fuses,  or  possibly  the  importance  of  f idol  i tv  between  t  ra in  in. 
and  testing  equipment  in  general. 

Three  hypotheses  were  of  particular  interest: 

1.  Students  trained  on  the  6883  simulator  me  th, 
test  station  are  equally  accurate  in  so! vine 
trouble-shooting  problems. 

2.  Students  trained  on  the  6883  simulator  and  tin  1 
test  station  are  equally  efficient  in  sil'-ing 
trouble-shooting  problems. 

3.  Students  trained  on  the  6883  simulator  and  L'u  nSS t 
test  station  operate  the  actual  test  station  with 
equal  proficiency. 

To  test  the  first  hypothesis,  a  two-way  analysis  of  variance 
was  performed  on  the  test  score  data  with  training  and  tc-tinr  modes 
as  the  independent  variables.  The  main  effects  of  both  vari  idles  wen 
nonsignificant  (F(  1 , 105)  <  1)  for  training  and  (l'(i,ll'>  .  09  ‘ . 

for  testing)  as  was  the  Interaction  (F(  1,105)  =  d.'.oi,  n  -  .id).  !iiu.- 
there  was  no  evidence  to  suggest  that  training  di  f  fotvnt  in  1  I  v  aftcC.c. 
accuracy  of  trouble-shooting. 


The  second  hypothesis,  regarding  problem-sol vi 
was  tested  by  submitting  the  total  completion  time  men 
analysis  of  variance  similar  to  that  for  total  scores, 
of  training  was  revealed  (F( 1,105)  <  1)  but  a  signific 
main  effect  ( F ( 1 , 105 )  =  9.312,  p  =  .003)  and  a  margi 
interaction  (F(l,105)  =  2.999,  p  =  .086)  were  found, 
less  of  training  mode,  students  took  longer  if  lestid 
than  if  tested  on  the  actual  6883  test  station;  this  o 
what  more  pronounced  for  actual  test  station  than  for 
students.  It  Is  interesting  to  note  that  this  pattern 
also  obtained  for  total  test  scores  as  summarized  abov 
effects  were  not  statistically  significant. 
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Based  upon  observation,  there  were  two  possible  factors  involved 
in  this  finding.  First,  the  simulator  was  slower  to  respond  to  student 
input  than  the  actual  6883  equipment,  and  thus,  testing  on  the  simulator 
resulted  in  slower  overall  completion  times,  even  for  those  familiar 
with  the  equipment.  Simulator-trained  students  who  were  tested  on  the 
actual  test  station  did  not  immediately  take  advantage  o'  the  slightly 
faster  test  station  as  did  actual  equipment-trained  students.  Second, 
because  of  this  difference,  the  actual  equipment-trained  students 
learned  to  respond  quickly.  Therefore,  when  transferred  to  the  simulator 
for  testing,  more  errors  (increased  time)  were  made  until  the  student 
became  familiar  with  the  difference  in  machine  responses.  This  impres¬ 
sion  was  not  always  consistent  wiLh  item  error  rates  since  mistakes 
that  were  corrected  prior  to  step  completion  were  not  counted  as  errors. 
For  instance,  if  a  student  entered  a  test  number  too  quickly  for  the 
machine  to  record  it  properly,  but  corrected  the  error  before  pressing 
"test  request,"  no  error  was  noted.  The  pattern  of  partial  test  times 
lends  some  credence  to  this  notion.  There  was  a  difference  of  7.89 
minutes  in  total  test  time  between  actual-trained  students  who  were 
tested  on  the  actual  and  simulated  equipment,  but  6.34  minutes  of  this 
time  had  already  accrued  by  the  19th  test  item. 

Hypothesis  if 3  required  an  examination  of  the  simple  main  effect 
of  training  at  the  actual  equipment  level  of  testing.  Completion  times 
for  this  actual/actual  versus  simulator/actual  comparison  did  not 
differ  significantly  ( F ( 1 , 105)  =  2.193,  p  =  .14)  while  there  was  a 
marginal  difference  in  total  test  scores  (F(l,105)  =  2.903,  p  =  .09). 

That  is,  actual-trained  students  scored  slightly  higher  on  actual 
equipment  testing  than  simulator-trained  students.  This  effect  is 
minor,  though,  and  may  be  the  result  of  machine-specific  experience 
rather  than  qualitative  differences  in  training.  It  seems  entirely 
plausible,  for  example,  that  students  would  be  more  nervous  being 
tested  on  the  real  test  station  if  their  only  experience  had  been  with 
a  simulator  where  errors  are  not  as  serious. 

A  brief  examination  of  the  degree  of  assistance  required  was 
carried  out  despite  this  measure's  lack  of  sensitivity.  The  group 
means  reported  in  Table  8  are  extremely  similar  and  a  chi  square 
analysis  showed  no  evidence  that  ratings  were  related  to  experimental 
conditions  (x‘  (6)  =  9.63,  p  =  .14). 

In  sum,  then,  simulator-  and  AET-trained  students  did  not  differ 
appreciably  witli  respect  to  overall  trouble-shooting  ability  as 
measured  by  the  practical  test.  A  very  slight  advantage  in  test  accu¬ 
racy  was  found  for  actual-  as  opposed  to  s i mu  1  at  or- 1 ra ined  students 
tested  on  the  actual  6883,  but  this  finding  was  not  mirrored  using 
completion  time  as  a  measure.  The  simulator  proved  to  be  a  somewhat 
slower  testing  device  which  caused  some  interference  for  actual-trained 
students . 
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Training,  Testing,  and  Specific  Troub 1 e- Shoo t ing  Tes t  Item  Ana lysis 

Table  9  includes  the  error  rates  for  each  of  the  29  items  on  the 
experimental  trouble-shooting  test  as  a  function  of  training  and  testing 
modes.  A  review  of  the  data  suggests  the  following  observations: 

•  Of  the  five  safety  errors  committed  (item  #1),  four  were 
made  by  simulator-trained  students  and  the  fifth  was  made 
during  a  simulator  testing  session.  This  finding  sug¬ 
gests  that  students  may  have  failed  to  perceive  the 
simulator  as  an  actual  test  station.  Special  efforts 
should  be  directed  toward  ensuring  that  safety  is  not 
compromised  by  this  apparent  difference  in  attitude. 

•  Seventeen  of  the  27  errors  involving  entering  test  num¬ 
bers  (items  #2,  #6,  and  #9)  were  committed  during 
simulator  testing.  This  was  most  likely  the  result  of 
the  simulator's  somewhat  slower  response  time  noted 
previously  or  a  lack  of  sensitivity  in  the  simulator 
pushbuttons . 

•  No  errors  were  made  on  item  # 5,  since  the  completion  of 
this  step  was  entirely  a  matter  of  automatic  sequencing. 

This  item  was  excluded  from  determination  of  the  total 
test  score. 

•  Students  had  particular  difficulty  with  item  #18 — 
rerunnning  the  test  by  returning  to  the  previous  "0- 
ending"  test  number.  The  higher  error  rates  for  students 
tested  on  the  simulator  again  suggest  that  the  poor 
response  of  the  equipment  may  be  responsible.  Both  the 
AET  and  simulator  had  a  test-repeat  option  button  which 
did  not  require  returning  to  the  previous  "0-ending" 
test  number.  Students  trained  on  the  actual  equipment 
often  selected  this  option,  which  works  but  is  not  pre¬ 
ferred  in  this  case.  This  was  recorded  as  an  error. 

•  Students  trained  on  the  actual  equipment  were  more  likely 
to  suspect  a  test  station  fault  and  were  better  able  to 
apply  the  appropriate  procedure  (items  #22-#25).  This 
difference  may  be  the  result  of  different  amounts  of 
experience  with  test  station  malfunctions  or  related 

to  the  notion  that  the  simulator  is  not  a  real  test  sta¬ 
tion  and,  therefore,  not  prone  to  malfunctions.  In  general, 
decoding  the  fault  isolation  proved  to  be  a  difficult  task 
for  students. 

•  As  items  #28  and  #29  indicated,  students  tested  on  the 
actual  equipment  did  not  offer  alternative  explanations 
for  the  test  station  failure.  This  was  related  to  the 
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fidelity  issue  since  the  lighted  fuse  holder,  visible 
on  the  actual  equipnent,  was  not  available  on  the 
simulator  until  very  late  in  the  testing  schedule. 

Thus,  actual  tested  students  saw  the  problem  more 
quickly  and  did  not  need  to  pose  alternatives.  Due  to 
instructor  involvement,  the  intent  of  this  item  was 
not  realized.  That  is,  regardless  of  a  correct  solu¬ 
tion,  students  were  required  to  be  asked  about  alter¬ 
natives  by  instructors,  but  were  not. 

Two  general  comments  might  be  made  with  respect  to  the  fore¬ 
going  observations.  First,  a  number  of  errors  and  differences  in 
error  rates  appear  attributable  to  differences  in  the  testing 
equipment.  Second,  and  more  important,  the  pattern  of  errors  suggested 
that  students  may  treat  the  simulator  as  something  less  than  a  real  test 
station.  This  has  important  implications  for  teaching  safety  and  test 
station  maintenance. 


User  Acceptance 

Another  element  of  this  evaluation  was  to  obtain  an  indication 
of  the  utility  and  adequacy  of  simulation  training  from  two  perspec¬ 
tives;  students  and  instructors.  The  results  of  the  survey  of  each 
group  are  discussed  below. 

Analysis  of  Student  Interview  Data 

A  two-person  testing  situation  had  originally  been  proposed  to 
assess  training  effects  on  student  attitudes  and  abilities  in  the 
absence  of  supervision.  Specifically,  the  following  hypothesis  was 
proposed:  personnel  trained  on  the  <5883  simulator  and  <5883  test  station 
are  equally  familiar  and  comfortable  in  operating  the  actual  lest 
station  without  supervision.  While  this  methodology  was  not  imple¬ 
mented  due  to  low  student  availability  and  logistical  problems,  an 
attempt  was  made  to  address  this  question  indirectly  via  student 
interviews.  Fightv-s  ix  mrcent  of  s  iim>  I  a  tor-t  ra  i  nod  and  percent  of 
actual-trained  students  reported  "feeling  comfortable"  operating  the 
equipment.  This  difference  between  groups  was  not  significant  ( \ '  (I)  = 
1.395,  p  =  .5).  In  general,  both  groups  ol  students  telt  that  they 
understood  the  equipment's  operation  (80  percent  vs.  88  percent)  and 
that  they  were  prepared  to  handle  t roub I  e-shoot i ng  tasks  in  the  field 
(58  percent  vs.  bb  percent).  Additionally,  there  were  no  differences 
in  student  attitudes  concerning  their  perceived  level  of  competence, 
opinions  about  the  b88  5  block,  or  much  i  no-spec  i  t  it'  prelerences  as  a 
function  of  training,  mode.  Despite  variability  of  training  associated 
with  AITf  and  the  simulator,  students  were  just  as  likely  to  sav  that 
the  simulator  or  actual  equipment  training,  was  routine  (  30  percent  and 
'V2  percent,  respectively).  This  latter  point  was  particularly  sur¬ 
prising  in  light  of  tlie  dramatic  difference  in  the  number  and  variety 
of  lessons  offered  in  the  two  training  modes. 


These  observations  suggest  that  student  attitudes  were  not  par¬ 
ticularly  relevant  to  the  issue  of  simulator  effectiveness.  Because 
the  experimental  manipulation  was  introduced  early  in  the  course  and 
for  a  short  time  (2-3  days),  students  simply  did  not  have  enough  ex¬ 
perience  to  evaluate  the  training  methods  used.  This  highlights  the 
critical  role  of  instructor  attitudes  about  the  use  of  simulators  in 
training  on  their  subsequent  effectiveness.  Since  students  had  not 
yet  developed  the  necessary  frame  of  reference,  they  were  most  likely 
to  rely  upon  the  experiences  and  opinions  of  their  instructors  to 
form  attitudes  toward  the  simulator  training  equipment. 

Analysis  of  Instructor  Survey  Data 

Twenty  F-lll  instructors  at  Lowry  ATC  were  surveyed  regarding 
their  teaching  experience,  field  experience,  attitudes  about  simulated 
training  in  general,  and  opinions  about  the  6883  3-dimensional  simula¬ 
tor  specifically.  The  results  of  the  questionnaire  are  shown  in 
Table  10. 

Only  three  of  the  instructors  surveyed  had  taught  6883  theory 
and  only  two  had  taught  6883  practical  during  1979.  The  3-dimensional 
simulator  was  used  exclusively  in  the  6883  block  of  instruction;  there¬ 
fore,  very  few  of  the  F-lll  instructors  had  any  contact  with  the  simu¬ 
lated  trainer  as  an  instructional  instrument.  In  fact,  only  six  in¬ 
structors  had  seen  a  demonstration  of  the  simulator,  and  only  one  had 
used  it  as  a  regular  part  of  his/her  teaching.  The  average  length  of 
time  as  an  instructor  at  Lowry  ATC  was  15.9  months,  although  the  range 
of  time  was  from  one  to  48  months. 

Only  limited  data  were  available  to  assess  the  impact  of  the 
use  of  the  simulator  on  training  methods.  The  six  instructors  who 
had  some  contact  with  the  equipment  commented  as  follows:  two  reported 
becoming  aware  of  weaknesses  in  the  course  materials  and  their  own 
teaching  methods;  three  claimed  to  have  changed  their  emphasis  on 
various  aspects  of  the  course  material;  and  one  respondent  indicated 
that  he  requested  that  changes  be  made  in  the  Plan  of  Instruction  (POI) 
for  other  block(s)  based  on  the  lesson  materials  utilized  on  the 
simulator.  Additionally,  instructors  commented  that  the  simulator  was 
too  slow  in  response  time,  lacked  well-designed  software,  and  was 
accompanied  by  poorly  organized  technical  manuals.  Instructors  did, 
however,  expect  that  simulators  would  replace  many  AETs  in  the  future. 


Estimation  of  Field  Preparedness 

Two  methods  were  used  to  assess  the  impact  of  training  mode  on 
student  acquisition  of  field-relevant  skills.  One  analysis  dealt  with 
the  results  of  the  Projected  Job  Proficiency  Test  (PJPT)  which  was 
administered  to  all  students  at  the  conclusion  of  the  6883  instruc¬ 
tional  block.  As  discussed  previously  in  this  report,  the  PJPT  was 
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TABLE  10 

Results  of  Lowry  ATC  F—  111  Instructor  Qi.es i ionn. lire 

Questions  Scale  f 

Simulator  training  is  a  good  idee.  2.3 

Simulator  training  can  be  more  effective  than  accual  equipment  training.  j.Q 

Simulator  training  can  provide  training  equivalent  with  actual  equip¬ 
ment  training.  2.7 

Simulated  trainers  must  be  highly  similar  to  actual  equipment  to  he 

effective.  2.0 

Simulated  trainers  can  provide  adequate  training  at  a  cost  savings.  2.9 

Simulated  training  allows  for  more  complexity  of  training.  2.9 

Simulated  trainers  are  more  reliable  than  actual  equipment.  2.2 

Simulated  trainers  teach  safety  procedures  better  than  actual  equipment.  2.8 

Simulated  trainers  provide  more  variety  of  training  than  actual 

equipment.  2.9 

Simulated  training  is  something  I  would  use  as  an  integral  part  of  nv 

teaching  program.  2.5 

Important  aspects  of  simulated  training  equipment  include: 

The  complexity  of  the  equipment.  2.5 

The  capability  of  the  software  to  meet  STS  and  Air  Force  objec¬ 
tives.  1.9 

A  lower  cost  of  hardware  and  operating  expenses  compared  to 

actual  equipment.  1.3 

A  high  degree  of  similarity  of  the  simulated  equipment  to  the 

actual  equipment.  1.6 

A  savings  in  the  time  required  for  training.  2.5 

The  degree  of  Air  Force  personnel  control  over  the  design  cf  the 

equipment.  1.9 

The  capability  of  Air  Force  personnel  to  modifv  existing  nr  to 

create  new  lessons  for  the  simulator.  1.8 

The  mobility  of  the  equipment.  5.2 

The  reliability  of  the  equipment.  1.4 

The  ease  of  maintenance  of  the  equipment.  1.5 

The  ability  to  monitor  student  performance.  1.6 

The  ease  of  use  for  the  instructional  staff  in  presenting  train¬ 
ing  materials.  1 . 8 

Responses  of  those  who  had  some  experience  with  the  3-2  simulator: 

The  hardware  is  too  simple  for  the  simulator  to  be  an  eiiective 

training  instrument.  3.5 

The  software  is  well  designed  for  Instructional  purposes.  2.5 

The  lessons  meet  STS  and  course  objectives.  2. 1 

Students  appear  to  regard  the  simulator  as  performing  as  an 

actual  test  station.  3.0 

The  simulator  Is  a  better  training  instrument  tlnn  the  actual 

test  station.  2.7 

1  have  used/will  use  the  3-D  simulator  js  an  integral  part  of  mv 

teaching  program.  2.s 

•The  scale  used  was  a  five-point  scale  whore  1  *  agree  strongly  and  5  -  dlsagrei 

strongly. 


specifically  developed  for  this  purpose  (rf.  p.  38).  Since  subse¬ 
quent  instructional  blocks  included  training  components  similar  to 
those  in  OJT,  student  performance  in  those  blocks  also  indicated 
field  preparedness.  Therefore,  the  second  analysis  examined  the 
results  of  AF-administered  achievement  tests  for  these  later  blocks. 

Projected  Job  Proficiency  as  a  Function  of  Training  Mode 

The  PJPT  was  a  paper-and-pencil  test  that  included  two  sec¬ 
tions:  32  short-answer  completion  items  (Part  I)  and  38  four-alterna¬ 
tive  choice  items  (Part  II).  Each  completion  item  was  worth  one  point 
toward  the  total  score,  although  partial  credit  was  awarded  for  some 
answers.  Table  II  summarizes  the  PJPT  results  as  a  function  of  6883 
training  and  includes  the  results  of  two  one-way  analyses  of  variance. 
The  small  sample  sizes  reflect  the  fact  that  this  test  was  developed 
subsequent  to  project  start-up. 


TABLE  11 

Field  Criterion  Performance  Test  Results  as  a  Function 
of  6883  Instructional  Training 


Training  Mode 

F- Ratio 

Actual 

Simulator 

Short  Answer 

N' 

45 

34 

(Part  I) 

X 

13.89 

13.99 

<1 

a 

5.46 

5.57 

Multiple  Choice 

N 

45 

34 

(Part  II) 

X 

23.91 

22.97 

1 

a 

3.65 

4.68 

The  primary  hypothesis  to  be  tested  with  this  measure  was  that 
students  trained  on  the  6883  simulator  and  the  6883  test  station  acquired 
equivalent  job-related  experience.  This  hypothesis  was  not  rejected 
when  PJPT  scores  were  the  dependent  measure.  However,  a  number  of  con¬ 
tributing  factors  may  have  made  this  measure  less  sensitive  than 
desired. 


In  general,  the  test  was  long  and  difficult.  This  was  parti¬ 
cularly  apparent  from  the  Part  I  (administered  after  Part  !l)  results, 
where  the  low  scores  typically  reflected  a  failure  to  finish  the  test 
within  the  three  hours  allotted.  Part  II  scores  were  also  low,  sug¬ 
gesting  that  the  test's  difficulty  may  have  had  a  discouraging  effect 
on  student  performance  from  the  outset.  Another  problem  was  that 
guidelines  were  not  made  explicit  with  respect  to  guessing  on  the  Part 
II  items.  Presumably,  some  students  did  select  their  best  guess  when 
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unsure  of  an  answer,  yet  the  fact  that  others  left  some  items  blank 
suggests  that  not  all  students  were  using  that  strategy. 

Subsequent  Classroom  Performance  as  a  Function  of  6883  Training 

Another  facet  of  assessing  possible  training  effects  on  job- 
related  abilities  was  the  analysis  of  subsequent  instructional 
block  scores.  It  was  anticipated  that  practical  blocks  would  be  parti¬ 
cularly  sensitive  to  simulator  versus  actual  training  experiences  in 
the  6883  block  and  also  most  similar  to  the  eventual  field  training 
situation.  Four  of  the  nine  subsequent  training  blocks  were  practical, 
but  no  training  differences  could  be  found  since  all  students  receive 
a  grade  of  "satisfactory"  in  each.  The  failure  of  this  data  to  detect 
any  differences  seems  attributable  to  two  methodological  constraints. 
First,  the  method  of  practical  block  evaluation  (satisfactory-unsatis¬ 
factory)  was  not  sufficiently  sensitive,  especially  for  such  a  homo¬ 
geneous  sample  of  students.  Second,  seven  students  were  deviated  out 
of  the  final  two  practical  blocks  and  all  seven  had  been  simulator- 
trained  on  the  6883. 

Since  practical  blocks  offered  no  useful  data,  performance  in 
subsequent  theory  instruction  blocks  was  examined  as  an  alternate  means 
of  assessment.  Table  12  presents  the  mean  block  scores  for  the  five 
theoretical  courses  as  a  function  of  6883  training  mode.  A  series  of 
one-way  analyses  of  variance  showed  no  training  differences  in  these 
scores  and  these  findings  were  reflected  bv  the  low  training  score 
correlations  reported.  In  brief,  then,  students  seem  to  perform  equally 
well  on  subsequent  instructional  blocks  whether  trained  on  the  6883  3- 
dimensional  simulator  or  the  6883  test  station 


Assessment  of  Field  Performance 

To  determine  whether  simulator-  and  net ua 1-t ra inod  technicians 
were  equally  capable  of  operating  F- 1 1 1  automatic  lest  stations  in  the 
field,  subjective  ratings  of  each  tochnicians's  performance  were  col¬ 
lected  from  both  the  technicians  themselves  and  their  supervisors. 

The  abilities  of  those  students  assigned  to  6883  and  6873  equipment  were 
of  particular  interest.  Since  the  impact  of  6881  training  mode  might 
be  expected  to  change  with  experience,  two  field  follow-up  surveys  wore 
conducted,  the  first  in  Sept ember /October  1979  and  the  second  in 
January  1980. 

Student  Follow-up  Surveys 

Table  13  presents  mean  technician  ratings  for  each  o!  the  12 
survey  questions  as  a  function  of  6883  ATI'  training  mode  ami  time  spent 
in  the  field.  Mighty-live  technicians  were  included  in  at  least  one 
fol low-up  and  66  were  included  in  both  surveys.  Although  116  students 
participated  in  the  DKT  evaluation,  a  number  of  them  could  not  he 


5 


TABLE  12 


Correlation  Coefficients  Relating  Training  to  Subsequent 
Instructional  Block  Performance 
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TABLE  13  (cont. 


Located  for  subsequent  field  assessment  ilu'e  to  attrition,  renss  i  guru  n  t  , 
or  inaccurate  personnel  records.  Also,  because  of  the  six  to  eight 
week  time  lap  between  ATC  graduation  anti  commencement  of  actual  field 
duties,  the  last  classes  in  the  DKI  sample  had  licit  vet  had  anv  !  ield 
experience  ;it  the  time  of  the  second  follow-up. 

The  data  in  Table  11  were  based  on  the  combined  responses  from 
both  follow-ups  so  that  a  maximum  of  I  a  1  ratines  were  obtainetl  per 
item.  When  distributed  across  the  16  training  by  experience  conditions, 
however,  cell  sample  sizes  varied  from  1  to  IS  observations.  because 
of  this  disparity  and  the  fact  that  66  technicians  were  surveyed  twice, 
the  data  are  presented  primarily  for  descriptive  purposes.  Matings 
were  fairly  consistent  within  items  and  appeared  generally  unrelated  to 
training  mode  or  amount  of  field  experience.  A  separate  cons i derat i on 
of  responses  provided  by  6881  and  6871  technicians  was  not  leasible 
because  of  the  small  number  of  respondents  assigned  to  those  test  sta¬ 
tions  (on.lv  14  in  the  first  and  Id  in  the  second  iol low-up). 

In  addition  to  the  questions  summarized  in  fable  1  1,  the  tech¬ 
nicians  were  also  asked  to  provide  comments  on  the  Alt',  course  structure 
and  the  relevancy  of  ATC  course  content  to  their  current  field 
assignments.  I  he  following  opinions  were  expressed: 

•  TO  familiarity  and  t  rouble-shoot  i  nc.  skills  were  cited 
as  the  most  useful  aspects  of  Alt’  training. 

•  Theory  portions  of  Alt',  training  were  considered  t  o  he 
of  limited  use  in  current  field  assignments. 

•  Most  respondents  felt  that  ATC  provided  adequate  prep¬ 
aration  for  0.1  f,  although  a  variety  ol  suggest  ions 
were  made  to  improve  the  training,  program.  Chose 
included  increasing  hands-on  IS  oper.it  ion  and  trouble¬ 
shooting,  paying  more  attention  to  paperwork  and  com¬ 
bining  theory  and  practical  portions  ot  instruction 

in  a  single  format. 

•  Concerning  the  6881  simulator,  recommondat ions  were 
made  to  increase  fidelity,  decrease  sensitivitv,  and 
increase  keyboard  response  speed.  Several  technicians 
felt  th, at  the  simulator  should  not  bo  used  as  part  ol 

l ra i n i ng. 

Supervisor  Follow-up  Surveys 

Table  14  presents  tile  mean  ratings  ol  toebnieiau  perl  o  man  ee 
as  determined  hv  field  supervisors  lor  <’881  training  nodes  ana  amount 
of  field  experience.  As  with  the  teehnieian  rat  ings  discussed  above, 
these  data  wore  based  on  the  eomb i ned  result',  ol  two  tel  lew-ups  and 
sample  sizes  varied  great  I v  between  eel  Is.  inptv  eel  Is  in  lable  I  ' 
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Responses  based  on  a  5-point  per t ormanee  scale  with  1  -  Very  Poor  and  5  -  Very  (ood.  Mean  ratings 
based  on  less  than  5  supervisor  responses  are  in  parentheses. 


ABLE  14  (cont. 


6883/6873  IS  AET  -  -  I  3.4  (3.7) 


TABLE 


Responses  based  on  a  5-point  performance  scale  with  1  “  Very  Poor  and  5  =  Very  Good.  Mean  ratings 
based  on  less  than  5  supervisor  responses  are  in  parentheses. 


resulted  since  supervisors  felt  they  were  not  qualified  to  rate  certain 
aspects  of  technician  performance  or  that  some  items  were  not  applicable 


No  noticeable  differences  in  technician  performance  were  found 
as  a  function  of  6883  training  mode.  Not  surprisingly,  however,  many 
items  did  suggest  an  increase  in  technician  abilities  with  increased 
field  experience. 


Cost  An  a Iv  s i s 

Table  15  shows  the  major  factors  of  the  life  cycle  cost 
comparison.  Recurring  and  nonrecurring  costs  have  been  combined  to 
provide  profiles  of  costs  in  constant  1978  dollars.  The  net  present 
value  of  each  cost  stream  projected  for  15-vear  life  cycles  and  dis¬ 
counted  at  10  percent  is  also  shown,  along  with  the  estimated  training 
costs  per  student-hour  of  instruction. 

TABI.K  15 

The  Life  Cycle  Cost  Comparison 


Cost  Categories 

■ - 

Si imi  lator 

ART 

Rac i 1 i t i es 

$  110,650 

$  110,630 

Kqui  pi. ten  t 

1,59.',,  330 

4,902,  1  40 

industrial  Materials 

26,000 

27,890 

l’ersonnc*  1 

89,270 

72.530 

St  udents 

357,770 

357,770 

Misee 1 l ancons 

0 

0 

TOTAL 

$2,  18  1,000 

$  5, 4 70, u SO 

NI’V  (1978) 

$1,501,090 

$  l,8‘t5,08(l 

Cost  Effectiveness — 

$s  pc  r  s  t  mien t -hour 

$  148/ s  t  udenl -hour 

h— - - : -  - - -  . 

$902 /s t  mien  I  -hour 

Table  16  and  IT  show  tabulations  of  the  individual  cost  esti¬ 
mates  for  elements  that  comprise  each  category,  together  with  explan¬ 
atory  information  for  the  ART  and  simulator,  respectively.  The  tables 
are  structured  so  that  the  1978  Investment  Costs  could  be  considered 
as  sunk  costs:  Trout  this  perspective,  the  costs  in  constant  dollars 
of  operating  the  ART  and  the  simulator  for  15-year  life  cvcles  arc 
$3,366,150  and  $1,588,020.  The  life  cvcle  cost  comparison,  which  also 
clearly  favors  the  simulator,  shows  that  simulation  is  an  important 
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TABLE  17 

Cycle  Costs  (Thousands  of  Dollars)  for  6883  Slmulalor--Cons tant  1978  Dollars 


w 

T3 

U 

(A  (A 

c 

"3  T> 

o 

•j 

>-  V. 

41 

lj  *-i 

cj 

O  O 

* 

m 

O  o 

“'*■>  •*-> 

X> 

c  o 

c 

l-  u 

c.  a. 

a 

E 

0> 

E 

E 

<S 

3 

c  c 

O 

o  0 

U  j 

r; 

O  /-N 

4-> 

> 

T3  ~a 

at  ?* 

w. 

0/  o 

c  — 

4-1 

tA  tA 

;  o 

~  TC 

JZ  c 

-Q  X> 

u  o 

1  V- 

•  a. 

O 

c; 

4-1  i-i 

a  t/l 

!  c 

v:  tr. 

c. 

! 

re  | 

rc 

rv» 

T3  0C 

cc 

o  c  . 

{  u 

o>  -c 

y  o  j* 

0  C  1 

>  3 

0 

i  sc  re  1 

*  L.O  ( 

U 

V.  "0  CJ  , 

i  U  *J 

4-0  • 

re 

>.  C  O'  , 

*-*  c 

c  c~  i 

V  O 

r-  -t-  </> 

SC  -J 

r«  *j 

o  *-> 

O  >“  4-> 

u  re 

•  c  re 

H 

CJ  Cl 

IA  3 

Cl 

o  s 

ere  tn 

4J  V- 

»~i  <A 

C  UJ  Jc 

Cl 

•  ©  -1 

CL 

>  c"' 

0  x> 

— .  -  O  c- 

•Ji  00 

r  fo  3  o 

vO 

,  C-  CJ 

o 

u:  </>  CJ 

1  52 

© 

! 

4A  j 

CJ  »- 

lA  © 

00 

LTV  © 

00  © 

J»  1 

O  i 

*“•  I 

© 

lA 

CJ  1 

CJ  -~ 

4A  O 

c->  ( 

1/1  O 

00  O 

O 

vA 

_ 

CJ  — 

ir.  |  © 

© 

4C  © 

co  1  © 

CJ 

i 

m 

1 

O 

lA  1 

O' 

c;  — 

4A  j  O  1 

© 

»A  © 

oc  ]  © 

CJ 

{  I 

© 

— '  o. 


—  CD 

N  O 

K 


n  cl 

o  cl 
•n  ~  < 


4ft.  ■ 


**.**?  ?» » 


training  option  for  maintenance  training.  The  following  examination  of 
some  of  the  cost  assumptions  inherent  in  this  comparison  only  supports 
this  finding. 

Cost  Assumptions 

A  major  issue  involved  in  planning  this  analysis  was  that  the 
simulator  system  cost  was  actually  known  to  he  higher  than  is  repre¬ 
sented  by  the  contract  cost  of  $548,000.  Only  now  can  this  be  put 
in  perspective.  First,  the  question  can  he  asked,  "What  is  the  impact 
on  the  cost-effectiveness  ratio  if  the  actual  cost  of  the  simulator  is 
perhaps  double  the  contract  price?"  In  a  net  present  value  (NTV)  cal¬ 
culation,  the  acquisition  costs  are  not  discounted,  thus  the  effect  of 
doubling  the  contract  price  is  not  obscured  by  the  discounting  process. 
If  the  actual  cost  of  the  simulator  were  doubled,  increasing  the  Nl’V 
of  the  simulator  to  $2,731,000,  the  ratio  of  cost  per  student-hour  of 
instruction  would  increase  only  25  percent  ($474/hour),  just  about  one- 
half  of  the  ratio  found  for  the  AFT. 

Second,  the  analysis  also  produced  some  evidence  that  Lhe  actual 
cost  of  the  simulator  was  approximately  double  the  contract  price.  The 
Logistics  Support  Cost  model  developed  by  the  Simulator  Sl’O  (1979) 
provided  insight  into  the  relationship  between  system  cost  and  the 
Sustaining  Investment  cost  component,  which  involves  test  station  spares 
and  spare  components  and  inventory  management.  This  model,  used  to 
assess  costs  of  the  F-16  simulator,  indicated  that  the  initial  system 
acquisition  and  support  costs  were  approximately  equal  to  the  15-vear 
life  cycle  costs  projected  for  "Sustaining  Investment"  ($4,919,000  and 
$4,965,000  respectively^).  This  relationship  is  important  because  it 
reflects  the  lower  cost  of  spares  associated  with  simulators  based  on 
actual  cost  experience  with  one-of-a-kind  systems.  Table  17  presents 
an  estimate  for  sustaining  investment  over  the  life  cycle  of  the  6883 
simulator.  The  $941,000  projected  is  consistent  witli  an  estimate  of 
$1,000,000  for  the  actual  system  and  support  costs  of  the 
simulator. 

In  considering  ttie  policy  implications  of  a  2  to  I  ratio  in  the 
cost  effectiveness  of  the  6883  simulator,  it  is  important  to  understand 
that  the  estimate  for  the  AFT  is  extremely  conservative.  It  was  noted 
earlier  that  the  costs  of  the  CFNI'AC  computers  were  not  allocated  to 
the  6883  test  station  and  examination  of  Table  16  shows  that  no  cost 
element  was  included  in  the  equipment  category  that  reflects  the  cost 
of  installation  and  start-up  of  the  AFT.  Installation  and  start-up 
costs  for  the  simulator  were  fully  allocated  and  it  can  be  seen  in 
Table  17  that  these  costs  were  perhaps  unexpectedly  high  due  to  prob¬ 
lems  encountered  in  bringing  the  simulator  on-line  during  1978  and  1979. 


"See  Table  1,  I. SC  Model  Fstimnte  of  the  Cost  of  the  ilardwan- 
Based  .Simulated  .Av  inn irs  hit  e  media  to  Shop  for  Maintenance  [rain¬ 

ing  (Kggemeier  et  a  1  . ,  1979). 
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The  particular  value  of  the  cost  comparison,  then,  lies  in  the  notion 
that  with  cost  experience  derived  from  actual  trainer  alternatives, 
there  is  an  obvious  preference  for  the  simulator. 

Alternative  Simulation  Scenarios 


A  2  to  1  cost  effectiveness  advantage  lias  been  determined  for 
the  6883  simulator  under  the  assumption  of  equal  training  ef feet i veness 
and  identical  patterns  of  trainer  use.  This  determination  has  particu¬ 
lar  significance  if,  in  fact,  the  simulator  does  provide  student  train¬ 
ing  to  the  same  level  as  achieved  on  the  AET.  As  already  discussed, 
while  no  major  performance  differences  were  observed,  it  is  not  true 
that  the  content  of  training  and  use  patterns  were  identical  for  all 
groups  studied.  The  following  discussion  presents  a  particular  per¬ 
spective  on  this  issue,  a  perspective  suggested  by  the  strong  cost 
advantages  of  the  6883  simulator. 

The  concern  about  training  effectiveness  of  the  simulator  can 
be  seen  in  figure  6.  The  diagram  indicates  that  the  simulator  effec¬ 
tively  trains  procedures  and  trouble-shooting  through  the  automatic 
testing  features  of  the  device  to  a  level  consistent  with  the  SI'S  for 
3-level  personnel.  The  broken  line  associated  with  the  AKT  indicates 
that  the  AET  trains  to  the  3-level  less  well  because  of  the  unpredict¬ 
able  nature  of  equipment  performance  during  each  training  session  (the 
uniformity  of  training  issue).  The  simulator  can  train  consistently 
to  the  3- level.  The  AET  can  do  something  that  cannot  be  done  on  the 
simulator,  however,  and  that  is  to  provide  hands-on  trouble-shoot ing 
experience  when  the  fault  cannot  be  corrected  through  automatic  testing 
procedures.  It  should  he  clear  from  the  diagram  that  if  the  technical 
school  is  to  provide  experience  in  the  manual  t rouble-shoot i ng  domain 
(regardless  of  how  well  such  requirements  are  ret  looted  in  the  STS), 
then  AETs  are  essential  to  the  training.  This  cone  1  us  ion  ignores  the 
potential  for  another  type  of  simulator  which  is  designed  to  provide 
manual  trouble-shooting  training.  With  this  hierarchy  ot  training 
objectives  in  mind,  we  invoke  the  cost  model  and  analysis  tor  guidance 
in  examining  alternatives  for  simulation  as  demonstrated  bv  the  6883 
dimensional  simulator. 

Tlte  generic  test  station  simulatin'.  This  first  scenario 
examined  considers  the  potential  for  utilizing  a  generic  lost  station 
simulator  as  a  means  lor  providing  structured  training,  in  l  rouble- 
shooting  procedures.  I  he  assumption  is  made  that  those  procedures 
employ/ only  automatic  testing;  lor  fault  isolation.  Ihe  presumption  is 
also  made  that  training  on  the  generic  simulator  would  eliminate  the 
need  for  AKT  training  on  some  of  the  ten  trainers  now  used  in  the  course, 
it  will  he  seen  that  this  scenario,  as  well  as  the  others  that  lollow, 
is  hampered  by  our  lack  ol  information  about  the  lilo  cvcle  cost  ol 
the  entire  training,  system,  but  certain  conclusions  can  be  drawn  based 
on  the  cost  ana  I  vs  is  ol  the  688  1  i-d i mens ionn I  simulator  and  All. 


Proficiency  of  5-level  airmen 


Table  18  reviews  the  operating  status  of  the  ten  test  stations 
as  of  August  31,  1979.  The  table  suggests  (1)  that  manual  trouble¬ 
shooting  experience  cannot,  be  a  feature  of  all  practical  exercise  ses¬ 
sions  due  to  variations  in  test  station  availability,  and  (3)  that  3- 
ditnensional  simulators  patterned  after  Lho  6883  could  be  useful  to 
deliver  training  related  to  procedure  and  automatic  fault  isolation, 
given  the  variability  of  test  station  status.  from  this  assessment,  it 
could  be  argued  that  Lest  stations  with  the  best  records  of  operational 
status  are  in  a  real  sense  now  being  used  as  generic  Lest  stations  —  to 
the  extent  that  experience  with  procedures  and  fault  isolation  must  be 
transferable  to  other  (nonoperating)  Lest  stations. 

The  pattern  of  use  for  the  generic  simulator  proposed  in  this 
scenario  is  one  in  which  training  is  tailored  in  response  to  current 
AET  status.  By  designing  a  library  of  lessons  to  reflect  different 
classes  of  faults  and  related  procedures  (reflecting  individual  test 
station  characteristics),  the  lessons  selected  for  particular  student 
trials  could  serve  to  help  link  the  experience  gained  on  the  simulator 
with  that  normally  obtained  on  the  nonfunctional  AET. 

The  question  of  the  cost  effectiveness  of  the  generic  simulation 
in  this  scenario  can  be  examined  in  a  straightforward  way.  Clearly, 
the  scenario  does  not  promise  cost  savings  because  the  simulator  repre¬ 
sents  additional  cost.  In  fact,  increased  training  costs  will  he 
incurred  because  the  pattern  of  simulator  use  proposed  does  not  disturb 
the  life  cycle  cost  picture  for  the  AETs.  In  a  life  cycle  cost  analy¬ 
sis  of  the  AET  training  system,  however,  if  the  costs  associated  with 
student  and  instructor  time  are  aggregated  for  all  procedural  and  fault 
isolation  training  time  lost  due  to  AET  status,  then  it  is  conceivable 
that  the  benefits  of  the  generic  simulator  will  exceed  costs.  If,  for 
example,  as  few  as  120  hours  per  student  are  lost  out  of  approximately 
930  hours  required  for  the  course  (13  percent),  then  a  simulator  would 
be  justified  on  the  basis  of  its  cost/benefit  ratio  over  the  15-year 
life  cycle  since  increased  training  effectiveness  would  offset  the 
higher  training  costs. 

Combined  test  station/simulator  complements.  This  second 
scenario  examines  the  situation  in  which  a  full  complement  of  simula¬ 
tors,  one  for  each  of  the  AETs,  is  postulated.  The  pattern  ol  use 
considered  is  one  in  which  the  simulator  is  used  to  train  procedures 
and  automatic  fault  isolation  and  the  AET  is  used  to  provide  for  equip¬ 
ment  familiarity  and  "manual"  trouble-shooting  experience.  The  sce¬ 
nario  cannot  be  evaluated  without  more  information  on  current  training, 
system  life  cycle  and  some  estimate  of  how  the  6883  AE'I  actually  com¬ 
pares  with  other  test  stations  in  order  to  gauge  simulation  costs. 

It  is  difficult  to  see  the  circumstances,  however,  in  which  the  combi¬ 
nation  of  reduced  training  time  and  reduced  AET  maintenance  costs, 
along  with  increased  manual  trouble-shooting  skills,  would  produce  a 
cost /belief  it  ratio  less  than  one.  The  or  I  v  other  major  source  ol  cost 
saving  that  has  the  potential  of  offsetting  the  simulator  investment 
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TABLE  18 

ATC  F— 111  LRUs  and  Operating  Status  as  of  8/31/79 


Test 

Station 

Number 

Description 

Operating 

Status 

6802 

Radar-Transmitter-Modulator  (RTM) : 

Models  A  &  E;  total  LRUs  possible  is 

4;  LRUs  available  at  ATC  are  MRT, 
Transmitter  Synchronizer 

Parts  and  maintenance 
problems;  limited  use 

6803 

Computer;  total  LRUs  possible  is  6, 

LRUs  available  at  ATC  are  Naviga¬ 
tions  Computer,  Feel  &  Trim 

None  at  al 1 ;  has  been 
down  2  months 

6805 

Attitude  and  Rate 

I. i mi  ted  use 

6815 

Video:  Models  A  &  E;  total  LRUs  pos¬ 
sible  is  6;  LRUs  available  at  ATC  are 
Antenna  Receiver,  Interference  Blanker 

Limited  use 

6863 

Digital;  total  LRUs  possible  is  5-6; 

LRUs  available  at  ATC  are  Computer, 

IRU 

Limited  use;  was  com¬ 
pletely  down  for  2 
years,  being  worked  on 

6882 

Radar- Transmi t ter-Modulator  (RTM) : 

Mode  1  D 

None  at  all 

6883 

Converter  Flight  Controls;  total  LRUs 
possible  is  4;  LRUs  available  at  ATC 
are  Feel  &  Trim  (869),  Feel  &  Trim 
(855),  Multiplex  Converter,  only  Yaw 
Computer  unavailable 

Best  running  TS; 
closest  to  realizing 
full  training  potential 

6885 

Video:  Model  D 

Limited  use 

6886 

•Electronic  Systems;  total  LRUs  possi¬ 
ble  is  over  80,  can  be  used  (with  adap¬ 
tation)  for  more  than  100  LRUs  in  the 
field;  LRUs  available  at  ATC  are  HSI, 

ECA  and  Antenna  Receiver  being  repaired 

Usually  operating,  but 
down  for  1  week 
recent  1 y 

6887 

Displays;  total  LRUs  possible  is  3-4; 
LRUs  available  at  ATC  are  2  (didn't  get 
names) 

Working  well 

CKNPAC 

Composed  of  3  separate  computers 

One  bit  by  lightning 
in  1978  is  still  down; 
other  two  are  working, 
but  with  intermittent 
prob 1 ems 
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is  to  return  some  of  the  AETs  (and  LKlls)  to  inventory  to  serve  as  spares 
for  the  flight  line.  Based  on  the  cost  effectiveness  ratio  found  for 
the  688'J  simulator,  it  would  be  necessary  to  return  perhaps  live  AETs 
to  F— Ill  weapon  system  inventory:  training  for  those  blocks  of  instruc¬ 
tion  affected  would  then  be  accomplished  on  T-d imens iona !  simulators. 
Without  attempting  to  generalize  further,  this  option  may  be  worthy  of 
consideration  by  ATC  given  the  existing  AET  operational  status. 

Replace  AETs  with  simulators.  This  final  scenario  is  poten¬ 
tially  the  most  revolutionary  one  because  it  suggests  new  patterns  in 
simulator  use  and  simulator  development.  It  proposes  that  all  of  the 
AE'I’s  are  returned  to  inventory  and  that  no  attempt  be  made  to  train 
manual  trouble-shooting  skills  on  actual  equipment  within  the  residence 
phase  of  F— 111  maintenance  training.  Manual  t roub 1  e-shoot i ng  princi¬ 
ples  could,  of  course,  be  taught  using  advanced  simulators.  There  are 
many  difficulties  inherent  in  this  scenario,  principally  because  imple¬ 
mentation  would  require  changes  both  in  the  residential  and  OJT  environ¬ 
ment,  as  well  as  changes  in  training  policy.  Tor  example,  the  residence 
training  environment  would  need  to  be  modified  to  optimize  the  eflec- 
tiveness  of  simulators  in  training  procedures  and  t roub I  e-shoot i ng, 
while  the  OJT  environment  would  need  to  he  organized  to  insure  that  the 
transfer  of  training  occurs  to  the  actual  equipment. 

The  scenario  postulates  that  the  course  duration  could  be 
reduced — possibly  to  16  weeks  instead  of  _M  weeks — in  view  of  the 
increased  training  potential  associated  with  the  simulators.  The  sce¬ 
nario  recognizes  that  students'  manual  t rouh 1  e-shoot i ng  skills  are  not 
well  developed  In  the  training  environment  and  that  simulation  tech¬ 
nology  could  lead  to  a  redefinition  ol  the  training  process. 

Evaluation  o I  this  scenario  is  not  possible  hero  because  ol  the 
broad  implications  for  OJT,  Logistics  Command,  and  ATC.  It  has  boon 
included  here  only  to  show  the  range  of  options  that  night  ho  consid¬ 
ered  in  an  attempt  to  establish  eost  effective  training  programs  that 
maximize  the  potential  of  I  us t rue t i on.  1 1  leennoiogv. 
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CHAPTER  (, 

GENERAL  DISCUSSION 


The  findings  of  this  study  suggest  a  number  of  critical  factors 
which  should  be  considered  in  future  efforts  to  investigate  the  role 
of  simulators  in  maintenance  training. 


Identification  of  Training  Objectives 

Any  comparative  analysis  of  simulators  and  AET  must  be  based  on 
a  clear  understanding  of  training  objectives.  The  present  study  was 
conducted  at  a  time  when  classroom  objectives  were  rapidly  changing  to 
meet  the  changing  requirements  of  the  field  assignment.  Civen  the 
lead  time  necessary  to  develop  the  specifications  of  a  simulator  and  to 
construct  the  trainer,  it  should  be  expected  that  the  capabilities  of 
the  simulator  were  somewhat  limited  with  respect  to  current  training 
needs.  Since  it  is  unlikely  and  perhaps  undesirable  that  the  objec¬ 
tives  of  a  training  course  can  be  stabilized  for  long  periods  of  time, 
the  useful  lifespan  of  major  simulators  may  be  significantly  increased 
by  emphasizing  training  related  to  the  general  skills  required  to 
operate  and  maintain  all  test  stations.  The  specialized  functions  of 
each  test  station  could  be  simulated  with  less  costly  module  simulators, 
which  are  either  disposable  or  reprogrammable  to  meet  changing  needs. 
Such  an  approach  is  not  unlike  the  "Test  Station  Replaceable  Unit"  (TRU) 
design  of  equipment  already  in  use.  While  the  main  purpose  of  the  TRUs 
is  to  improve  the  capability  of  repairing  test  stations  quickly,  there¬ 
by  reducing  "downtime,"  the  component  system  also  makes  it  easier  to 
update  (within  limits)  test  stations  to  provide  testing  capability  for 
new  or  modified  aircraft  systems.  There  is  no  reason  to  expect  that  a 
simulator  designed  to  replace  a  test  station  will  not  be  subject  to  the 
same  limitations  on  utility  due  to  improvement  in  aircraft  design. 

In  view  of  changing  course  objectives  and  needs  in  the  field, 
it  seems  that  an  explicit  statement  of  minimal  trouble-shooting  stan¬ 
dards  should  be  made.  These  standards  could  then  form  the  basis  of  a 
variety  of  training  strategies  and  trainers. 


Define  the  Anticipated  Role  of  Simulation 

From  the  outset  of  any  investigation,  the  intent  of  introducing 
simulators  into  training  should  be  clearly  stated.  That  is,  it  should 
be  determined  if  the  simulator  designed  is  expected  to  }\plae,  or 
supplement  actual  training  equipment.  Botli  are  valid  approaches  to  the 
use  of  simulation,  of  course,  but  require  somewhat  different  equipment 
and  research  designs. 
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If  the  primary  objective  of  incorporating  a  simulator  into 
training  is  to  replace  more  costly,  less  reliable,  anc]  more  dangerous 
actual  equipment,  then  it  is  clear  that  both  psychological  and  physical 
fidelity  must  be  considered.  In  this  case,  it  is  more  likely  that  the 
simulator  will  include  the  capability  of  demonstrating  basic  operations 
procedures  while  duplicating,  to  a  large  extent,  the  physical  appearance 
of  the  actual  equipment.  Physical  fidelity  is  important  since  trainees 
will  be  assigned  to  actual  equipment  in  the  field.  A  lack  of  sufficient 
physical  fidelity  in  the  simulator  and  no  exposure  to  actual  equipment 
in  training  would  almost  certainly  result  in  reduced  initial  perform¬ 
ance  on  actual  equipment  in  the  field.  However,  research  may  determine 
which  tasks  require  full  fidelity  trainers. 

If,  on  the  other  hand,  the  objective  of  incorporating  a  simula¬ 
tor  into  training  is  to  supplement  the  use  of  actual  test  station 
equipment,  then  psychological  fidelity  should  be  emphasized.  In  either 
case,  psychological  fidelity  of  sensitivity  to  operator  actions  must 
be  established.  Interviews  with  students  who  had  contact  with  the  6881 
simulator,  conducted  directly  after  the  6883/6886  practical  block,  and 
responses  from  technicians  on  the  field  follow-up  quest ionniares ,  indi¬ 
cated  disappointment  with  the  simulator  on  this  aspect  of  design.  That 
is,  while  the  two  trainers  looked  similar,  the  simulator  reacted  dif¬ 
ferently  (slower)  to  operator  input.  This  flaw  was  also  referred  to 
by  Becar  (1978)  in  his  evaluation  of  the  6883  Maintenance  Trainer  System. 
Supplemental  simulators  can  more  easily  focus  on  more  complex  tasks, 
including  training  on  equipment  malfunctions  which  cannot  be  introduced 
or  experienced  on  actual  test  station  equipment.  The  next  phase  of 
this  project,  the  evaluation  of  the  flat  panel  simulator  is  expected  to 
provide  data  on  this  issue,  since  the  simulator  includes  a  trouble¬ 
shooting  trainer. 

The  use  of  simulators  for  training  maintenance  skills  offers  an 
opportunity  to  provide  consistent  training  since  they  are  less  subject 
to  random  malfunctions.  Further,  simulators  designed  to  augment  ACT 
can  be  more  easily  used  to  train  personnel  in  the  operation  and  main¬ 
tenance  of  test  stations  in  general.  More  general  skills  (e.g.,  sys¬ 
tems  and  problem  solving)  not  unique  to  any  specific  test  station  can 
be  provided  as  an  introduction  to  training  on  specific  test  stations. 

In  the  present  study,  it  was  found  that  students  trained  on  the  6883 
3-dimensional  simulator  performed  as  well  as  students  trained  on  actual 
equipment;  the  true  training  benefits  of  the  simulator  were  probably 
not  realized,  however,  because  the  simulator  was  designed  to  rep  1 acc 
the  AET.  Improved  student  performance  was  expected  because  the  simu¬ 
lator  would  provide  more  consistent  training  experiences,  (liven  the 
underlying  assumption  that  the  6883  simulator  might  replace  ACT,  the 
outcome  of  the  cost  comparison  between  trainers  becomes  the  major 
factor  in  future  procurement  decisions,  given  approximately  equivalent 
training  capability.  Given  a  "supplemental"  objective,  improved  per¬ 
formance  becomes  the  major  factor  considered  and  an  analysis  of  cost- 
benefit  is  most  appropriate. 


Approach  to  Teaching 

It  is  unlikely  that  a  simulator  of  any  quality  will  be  accepted 
into  existing  training  curriculum  if  it  is  not  somewhat  consistent  with 
existing  student-teacher  interaction  policies.  It  seems  that  only  a 
self-instruction  mode  in  which  the  simulator  guides  the  student  through 
a  series  of  problems  might  result  in  alteration  of  these  policies.  To 
encourage  teacher  acceptance,  the  simulator  should  be  effective  as  both 
a  visual  aid  and  demonstration  tool.  This  would  allow  the  simulator  to 
be  effectively  incorporated  into  training  segments  (e.g.,  theory  famil¬ 
iarization)  which  do  not  include  extensive  practical  trouble-shooting 
experience.  Such  a  dual  purpose  simulator  would  be  almost  essential  if 
replacement  of  existing  equipment  is  planned.  Also,  the  thorough  intro¬ 
duction  of  the  training  simulator  to  the  training  staff,  highlighting 
uses,  real  and  potential,  of  the  equipment  in  the  overall  training 
program,  will  improve  instructor  acceptance. 

The  environmental  constraints  on  this  study  suggest  that  many 
instructional  practices  have  evolved  which  are  deemed  necessary  to 
maximize  the  effectiveness  of  actual  equipment  as  trainers.  Given  this 
situation,  it  is  not  surprising  that  performance  differences  as  a  func¬ 
tion  of  training  equipment  were  not  observed.  The  potential  impact  of 
simulator  training  on  student  performance  may  be  realized  only  if  a 
utilization  strategy  accompanies  the  placement  of  a  simulator  into  an 
existing  training  environment.  This  plan  for  using  the  simulator  would 
insure  that  its  unique  training  capabilities  were  tapped,  and  benefits 
in  terms  of  improved  performance,  consistent  training,  reduced  training 
time,  and  cost  savings  might  then  be  measurable. 


Generalization  of  Findings 

As  evidenced  by  the  fact  that  an  entire  chapter  of  this  report 
was  devoted  to  the  impact  of  the  environment  on  the  evaluation  effort, 
the  generalizability  of  findings  can  only,  by  necessity,  be  limited. 
While  every  effort  was  made  to  adapt  experimental  design  principles  for 
use  in  this  natural  experiment,  it  was  not  possible  to  rely  on  many  of 
the  premises  of  basic  learning  theory.  Until  parameters  such  as 
content,  method,  and  duration  of  training,  all  known  to  affect  learning, 
are  subject  to  more  careful  control,  a  true  cost-effectiveness  analysis 
of  simulation  training  is  not  possible.  The  point  at  which  simulators 
provide  the  best  training  for  a  specified  cost  or  the  least  cost  for 
specific  training  can  be  determined  only  if  control  over  relevant 
learning  factors  is  possible.  To  answer  the  question,  "Do  simulators 
provide  more  cost-effective  training  than  AET?"  we  must  be  able  to 
maximize  the  use  of  simulator  capabilities  beyond  those  available  on 
actual  test  stations.  Simply  stated,  AETs  are  not  designed  for  train¬ 
ing  purposes;  simulators  can  be  designed  solely  for  that  purpose. 


Experimentation  in  the  natural  training  environment  is  neces¬ 
sarily  limited  by  the  fact  that  simulators  must  provide  training  at 
least  equivalent  to  that  provided  by  existing  AETs .  The  adequacy  of 
training  on  AETs  in  relation  to  the  STS  is  assumed  here;  however,  the 
issue  could  also  be  investigated  empirically.  Obviously,  it  is  not 
feasible  to  risk  the  possibility  of  providing  inferior  training  in  the 
interest  of  research,  which  is  focused  on  defining  the  conditions  of 
maximum  cost-effective  simulator  training.  There  are  two  viable  solu¬ 
tions  to  avoiding  the  limitations  imposed  by  the  natural  environment. 
First,  the  student  could  be  deviated  from  normal  training  to  partici¬ 
pate  in  a  well  controlled  research  study  and  then  subsequently  re-enter 
the  training  sequence  at  the  point  of  departure.  The  additional  cost 
of  deviating  students  (i.e.,  cost  of  additional  day  in  ATC)  would  be 
part  of  the  research  costs.  The  disruption  of  student  flow  from  the 
field's  perspective  would  occur  only  at  the  start  of  such  a  project 
and  should  not  cause  any  significant  shortages  of  field  personnel 
since  student  flow  is  normally  somewhat  erratic.  The  second  alterna¬ 
tive  is  to  carefully  structure  a  significant  block  of  training  time  to 
allow  the  research  project  to  be  integrated  into  the  existing  training 
sequence.  Students  would  complete  training  in  the  same  time  frame 
as  usual  (or  sooner) ,  and  any  adverse  impacts  of  the  research  on 
performance  could  be  corrected  by  CUT.  The  additional  training 
required  (if  any)  would  become  itself  a  measure  of  training 
effectiveness.  While  other  alternatives  may  be  possible,  the  main 
point  is  that  true  potential  of  simulation  in  training  can  be  deter¬ 
mined  only  by  a  focused  research  effort.  Clearly,  some  additional  costs 
will  be  incurred  by  such  a  research  effort — a  small  price,  however, 
given  the  potential  utility  of  the  information  in  defining  the  future 
role  of  simulation  in  maintenance  training.  Such  research  would  insure 
that  future  investments  in  simulators  would  he  based  on  factual  infor¬ 
mation  rather  than  assumptions.  The  information  obtained  should  high¬ 
light  the  conditions  under  which  the  use  of  simulators  in  maintenance 
training  is  most  effective.  The  overall  cost  savings  to  the  Air  Force 
would  he  extensive  regardless  of  the  findings.  Many  future  research 
dollars  could  he  more  wisely  invested  if  simulation  in  training  was  not 
demonstrated  to  be  cost-effective  even  under  ideal  conditions,  or  many 
training  dollars  might  be  saved  by  appropriate  utilization  of  simula¬ 
tion  in  training  as  defined  by  experimentally  determined  guidelines. 
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Section  A-l 


Student  Name 
Observer  _ 

Note  Time 
0 


TROUBLE-SHOOTING  TEST  AND 
DATA  RECORDING  SHEET 


S teps  Completed 


Training  Mode _ _ — _ - - 

Testing  Mode  - - - 

ERROR  NOTED  (indica 
Ves  No  step  II  and  your  comm 


1. 

Remove  all  jewelry. 

— 

— 

2. 

Enter  and  request  test  #330401. 

— 

— 

3. 

Verify  test  number,  press  START- 

- - 

— 

4. 

Verify  test  date,  press  START. 

— 

— 

5. 

Does  LRU  pass  test  #300130. 

_ 

— 

6. 

Testing  halts  at  #300610;  enter 
#300630,  TEST  REQUEST,  START. 

— 

— 

7. 

Testing  halts  at  #300755;  when  ' 
press  START,  observe  increasing 
voltage,  press  START. 

’go' 

+ 

_ 

8- 

Testing  halts  at  #300764;  when 
press  START,  observe  increasing 
voltage,  press  START. 

’go" 

... 

9. 

Testing  halts  at  #300824,  enter 
#300830,  TEST  REQUEST  START. 

_ 

— 

10. 

Testing  halts  at  #301752;  rotate  0SCP, 

INPUT  SIGNAL  SEL  switch  to  "digital 
multr.  ’  - 

— 

11. 

Press  NEC  DC  COUPLING  switch. 

_ - 

12. 

Press  P0S  DC  COUPLING  switch. 

— 

-- 

13. 

Set  VERNIER  to  "fully  cw." 

— 
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Note  Time  Steps  Completed  Yes 

14.  Set  V/DIV  at  .5.  _ 

15.  Set  SWEEP  MODE  on  "auto".  _ 

Note.  S  imul.  only:  P.eset  system  to 
allow  rerun  of  test  tape.  On  student 
console,  press  FAULT  DETECT /RETURN.  _ 

16.  Adjust  to  obtain  display,  observe 

sine  wave  (should  not.  be  there).  _ 


17.  Determine  that  the  Yaw  Board  TB3 
is  defective. 

18.  Rerun  test  tane  from  #301740. 

19.  Advise  board  replacement. 


20. 

21.  Student  should  offer  alternative 
explanations  of  malfunction. 
(Instructor  should  only  state  that 
"Yaw  Board  TB3  was  apparently  not 
the  cause.") 


22.  After  5  minutes:  Instructor  suggests 
that  "test  station  caused  the  mal¬ 
function  . " 

23.  After  another  5  minutes:  Instructor 
suggests  that  "the  test  program  should 
be  decoded." 

24.  Student  decodes  program  in  total. 

23.  Student  decodes  program  in  part. 

26.  Student  identifies  missing  power 
Input. 

27.  Student  notes  that  power  light  is  off. 


Note:  Simul.  only:  Indicate  TB3 
is  replaced.  On  student  console,, 
press  .START. _  1 

Rerun  test  tape  from  #301740. 


yes  No  Error  Noted 


28. 


29. 


30. 


Student  suggests  alternative  expla¬ 
nations  for  lack  of  power. 

Students  suggests  that  a  fuse  has 
blown. 

Degree  of  instructor  assistance 
required: 

very  little 
moderate 
a  lot 


Page  3  o 
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Section  A-2 

INTERVIEW  QUESTIONS  FOR  AIR  FORCE 
AVIONICS  TECHNICIAN  STUDENTS 


Did  you  feel  that  you  had  sufficient  training  time  on  tin  oqiii;\-.ont  1:1  i! 

6883  block? 

Did  you  feel  your  time  on  the  training  equipment  in  the  6883  M  vk  wt* 

well  utilized? 

What  segments  of  your  time  on  the  equipment  would  you  increase?  Di-cnusi 

Did  you  receive  sufficient  individual  attention  on  the  equipment  duri 

your  training? 

Was  there  variety  in  the  training  on  the  6883  equipment  or  was  it  rostlv 

routine? 

Do  you  feel  comfortable  operating  the  equipment? 

Do  you  feel  you  understand  how  to  operate  the  equipment  ?  i’ia  u  h.iv*  c. 

problems? 

Was  the  equipment  more  sophisticated  than  you  expected’  l.es-i? 

Was  the  equipment  more  sophisticated  than  vou  think  was  m-a  ■>  •  ir  ••  t  »  >  '  . 
your  training? 

Were  there  equipment  malfunctions  during  your  training  wiiiih  leic: 
training?  Benefited  your  training.’ 

Did  you  receive  trouble-shoot ing  experience  on  the  688?  ti.ur.i',  .  ..in"-.:; 

Would  you  like  to  have  more  trouble-shooting  experience  in  vein  i:uini  .. 

Do  you  feel  adequately  prepared  in  vour  t rouble-shoot ing  expcri.nct  :  r 
your  field  assignment? 

Has  your  training  given  you  sufficient  experience  in  u-  i:  t?;.-  ;  ‘  r 

to  solve  test  station  and  LRU  problems  in  the  field.’ 

What  would  you  change  in  the  TOs  to  make  them  more  u^eiui 

Did  you  find  the  TOs  adequate  for  solving  test  station  nd  I.M'  pi  !  ;  ! 

operating  the  equipment.’ 

Was  your  training  instructor  helpful  in  explaining  the  equip:-*  •  o  d  ;  t 

Did  the  inst rue tor  provide  sufficient  guidance  while  vou  were  ? •  »i:iinc  .  u 
the  6883  equipment?  Too  little?  Too  much? 

Was  the  instructor's  guidance  necessarv  to  surcosstut  complet ion  .o  v  > . : : 
training  on  the  6881  equipment? 

Did  the  instructor's  attitude  toward  the  training  equip-eu'  l  av.-  aav  ct*. 
on  your  training  on  the  equipment? 


21.  Did  the  testing  procedure  test  your  capabilities  on  the  training 

equipment? 

22.  Were  the  testing  procedures  difficult?  Why? 

23.  What  would  you  include  in  a  test  of  vour  capabilities  on  the  688 '}  training 
equipment?  Leave  out? 

24.  What  capabilities  should  a  1-level  avionics  technician  have  with  respect  to 
operating  test  station  equipment  in  the  field? 

25.  Do  you  feel  you  have  those  3- level  capabilities  as  a  result  of  your  training 

26.  What  factors  would  you  add  to  your  training,  or  increase  the  amount  of  atten 
tion  to,  to  better  prepare  you  for  your  field  assignment? 

27.  Do  you  understand  the  overall  purpose  of  the  training  program?  What  is  that 
purpose  to  your  knowledge? 

28.  Did  your  training  on  the  6883  equipment  help  your  overall  understanding  of 
the  purpose  of  the  training  program: 

29.  Do  you  have  any  additional  comments,  questions,  or  suggestions? 
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Section  A-3 


6883  3-DIMENSIONAL  SIMULATOR  EVALUATION  PROJECT 
INSTRUCTOR  QUESTIONNAIRE 


About  how  long  in  months  have  you  been  an  instructor  at  Lowry  AFB? _ 

Approximately  how  many  times  in  the  past  year  have  you  taught  each  or  the 
following  blocks? 

6833  Theory  _  6386  Theory _ 

6883  Practical _  6886  Practical _ 


Other  blocks _ 

If  you  have  had  field  experience  with  Test  Stations,  please  indicate  the 
length  of  such  experience  and  whether  your  experience  involved  TS  operation 

TS  maintenance,  or  both _ ____ _ 


4 

! 


In  the  next  section,  we  would  like  you  to  rate  how  much  you  would  agree  with 
each  of  the  following  statements  on  a  five(5)-point  scale,  where  l*agree 
strongly  and  5*disaeree  strongly. 


From  your  general  knowledge  of  and  experience  with  simulated 

feel  that  simulated  training:  agree 

strongly 

a)  Is  a  good  idea  1  2 

b)  can  be  more  effective  than  actual  equipment  1  2 


c)  can  provide  equivalent  training  with 

actual  equipment 

d)  must  be  highly  similar  to  actual  equipment 

to  be  useful 

e)  can  provide  adequate  training  at  a  cost 

savings 

f)  allows  for  more  complexity  of  training 

g)  Is  more  reliable  than  actual  equipncnt 

h)  teaches  safety  training  better  than 

actual  equipment 

1)  provides  more  variety  of  training  than 
actual  equipment 

j)  is  something  you  would  use  as  an  Integral 
part  of  your  teaching  program 


1  2 

1  2 

1  2 

1  2 

I  2 

1  2 

1  2 

1  2 


disagree 

strongly 

3  4  3 

3  4  5 

3  4  5 

3  4  5 

3  4  5 

3  4  5 

3  4  5 

3  4  5 

3  4  5 

3  4  5 


Please  rate  how  Important  you  feel  each  of  the  following  factors  is 


in  evaluating  a  simulated  training  device: 

a)  complexity  of  the  equipment 

very 

important 

1 

2 

3 

A 

unim¬ 

portant 

5 

b)  capability  of  the  software  to  meet  STS 

1 

2 

3 

A 

5 

and  Air  Force  objectives 

c)  a  lower  cost  of  hardware  and  operating 

expenditures  compared  to  actual  equipment 

1 

2 

3 

A 

5 

d)  a  high  degree  of  similarity  of  the  simul¬ 
ated  equipment  to  the  actual  equipment 

1 

2 

3 

A 

5 

e)  a  savings  in  the  amount  of  time  required 
for  training 

1 

2 

3 

A 

5 

f)  the  degree  of  control  of  AF  personnel  over 
the  design  pf  the  equipment 

1 

2 

3 

A 

5 

g)  the  capability  of  AF  personnel  to  modify 
existing  or  create  new  lessons  for  the 
simulated  trainer 

1 

2 

3 

A 

5 

h)  mobility  of  the  equipment,  for  versatility 
of  use  in  the  classroom 

1 

2 

3 

A 

5 

i)  reliability  of  performance  of  the  equipment 

1 

2 

3 

A 

5 

J)  ease  of  maintenance  of  the  equipment 

1 

2 

3 

A 

5 

k)  ability  to  more  closely  monitor  student 
performance  on  che  equipment 

1 

2 

3 

A 

5 

1)  variety  of  material  covered  in  lessons 
compared  to  actual  equipment 

1 

2 

3 

A 

5 

d)  ease  of  use  for  training  staff  in  present¬ 
ing  training  materials 

1 

2 

3 

A 

5 

Please  Indicate  the  amount  of  experience  you  have  had  with  the  6383  3-dimensional 
Training  Simulator  by  checking  the  appropriate  statements  below: 

a)  _ have  heard  about  it,  but  never  actually  used  it 

b)  _ have  seen  a  demonstration  of  it 

c)  _ have  had  limited  use  of  it,  as  a  reference  for  teaching 

d)  _ have  used  it  as  a  regular  part  of  ny  teaching 

e)  _ have  been  involved  with  writing  lessons  for  use  on  it 

f)  _ have  been  involved  with  the  design  and  development  of  the  unit 

g)  _ no  contact  or  experience  with  the  unit  (if  yes,  please  skip  the  next  section) 
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From  your  experience  with  Che  6883  3-D  Training  Simulator,  to  what  extent 


agree 

strongly 


disagree 

strongly 


would  you  agree  with  the  following  statements: 

a)  the  hardware  is  too  simple  for  it  to  be 

an  effective  training  instrument  1234 

b)  the  software  (lessons)  is  well-designed 

instruction  purposes  1234 

c)  Che  lessons  meet  STS  and  course  objectives  1234 

d)  students  appear  to  regard  the  simulator  as 

an  actual  Test  Station  1234 

e)  the  simulator  is  a  better  training 

instrument  than  the  test  station  1234 

f)  I  have  used/will  use  the  3-D  sim¬ 

ulated  trainer  as  an  integral  part  of  mv 

teaching  program  1234 


What  involvement  have  you  had  with  the  Denver  Research  Institute's  evaluation 
of  the  3-dimenstonal  simulator?  Please  check  any  applicable  statements. 

a)  _ proccored  the  three-hour  written  test 

b)  proctored  the  practical  performance  test 

c)  _ assisted  with  the  design  of  the  two  tests 

d)  _ was  interviewed  regarding  my  teaching  methods  and  course  material 

e)  _ had  no  involvement  with  the  DPI  evaluation  program  or  development  of 

materials  (If  no  involvement ,  shin  next  section) 

To  what  extent  would  you  say  your  involvement  with  the  DRI  evaluation  has 
influenced  your  approach  to  teaching?  Chech  any  appropriate  statements. 

a)  _ I  have  become  aware  of  weaknesses  In  nv  methods  of  presenting 

block  material 

b)  _  _ X  have  become  aware  of  weaknesses  in  the  course  material 

c)  _ I  have  rut  more  emphasis  on  certain  arejs  ot  the  course  material 

d)  _  _ I  have  made  changes  in  the  Plan  of  Instruction  (POI)  for  the  hleck(s) 

I  teach 

e)  _ I  have  requested  changes  in  the  course  material  for  the  block(s)  1 

teach 


f)  _ I  have  not  been  influenced  in  my  methods  of  teaching  bv  the  DPI 

evaluation 

g)  _  _ I  have  not  seen  any  reason  to  alter  course  materials  due  to  the 

DRI  evaluation 
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Do  you  have  any  additional  comments  to  make  regarding  the  6883  3-D 
Training  Simulator,  its  design  and  use,  or  the  DRI  evaluation  program? 


Section  A-4 


DO  NOT  WRITE  IN  THIS  BOOKLET 


FIELD  PERSONNEL  TEST  QUESTIONS 

for 

THREE-LEVEL  AVIONICS  TECHNICIANS 


Compiled  by 

The  Denver  Research  Institute 
for 

AF  Contract  No.  F33615-78-C-0018 
Project  No.  2381 
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DO  WOT  WRITE  III  THIS  BOOKLET 


FIELD  TEST  QUESTIONS 


PART  I 

1)  Which  TO  would  be  used  to  operate  the  Converter-Flight  Control 
Systems  Test  Station? 

2)  Which  TO  is  used  to  perform  Confidence  Testing? 

3)  Which  TO  contains  Maintenance  Test  Instructions? 

A)  Which  TO  would  be  used  when  ordering  parts  for  the  Signal  Converter 
Unit  Controller  A2A1? 

5)  Which  TO  would  be  used  when  ordering  parts  for  the  Signal  Data 
Converter  A2A8? 

6)  List  the  TO,  Section,  and  Paragraph  of  the  procedure  used  to 
calibrate  the  Variable  Power  Control  A1A5. 

7)  List  the  six  TOs  that  would  be  used  to  operate,  verify,  calibrate, 
trouble-shoot,  repair,  and  order  parts  for  the  A1  Converter  in  the 
Signal  Data  Converter  A2A8. 

Answer  the  following  questions  and  write  the  TO  reference  where  the 

answers  may  be  found. 

8)  What  happens  when  Address  E613  is  programmed? 


9)  What  is  produced  when  the  DATAC  Mode  Switch  is  rotated  through  STOP? 


10)  When  connecting  power  to  the  test  station,  what  is  connected  to  the 
red  lead? 


11)  Test  Point  11  on  the  D1U,  A2A5  is  used  to  monitor 


12)  What  is  the  part  number  of  an  NG86? 


13)  At  what  temperature  should  the  overtemp  lamp  in  rack  1  flash? 


14)  What  is  the  purpose  of  Signal  Select  Switch,  A3A2A31S2  when  in 
AUTO  TEST  POINT? 


I 

I 


IS)  DATAC  displays  DMM  BAL.  This  indicates? 


**&****■, W*?W.V’ 


**  *****  t~* 


16)  When  using  the  Oscilloscope  to  check  a  signal  of  35KKz,  which 
position  of  the  Main  Trigger  Coupling  Switch  should  be  used? 


17)  List  the  sequence  and  control  settings  to  light  the  Manual  Mode 
lamp  on  the  Power  Supply  Controller,  A1A2. 


18)  What  Address/subaddress  is  used  to  reset  the  Power  Supply  Con¬ 
troller,  A1A2? 


19)  What  Address/subaddress  is  used  to  reset  the  power  stimulus 
relays? 


20)  List  the  sequence  and  control  settings  and  all  the  indicated 
results  when  the  DATAC  Logic  Verify  is  performed. 


21)  List  three  conditions  that  will  light  the  Switch  Fail  indication 
on  the  DATAC  Prog  Stop  indicator. 


Decode  Test  Number  202574  of  Test  Tape  Request  Number  8399Q2  and 
answer  the  following  questions. 

22)  Describe  the  function  of  each  part  of  a  program. 

23)  What  is  the  part  number  of  the  Self-Test  Adapter? 

24)  What  is  the  value  of  the  measured  stimulus? 

25)  List  the  relays  by  reference  designation. 

(a)  Test  Point  Selection 

(b)  Test  Point  Distribution 

(c)  Stimulus 

26)  Identify  the  Patch  Panel  pin  numbers  used  to  route  the  signal 
through  the  Self-Test  Adapter. 
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27)  For  Test  Number  202574,  DATAC  Measurement  and  Result  displays 
F00003  and  NO-CO  J.ov: 

(a)  Why  Is  Test  Number  211307  displayed? 

(b)  What  Is  the  most  probable  fault? 

(c)  What  Is  the  second  most  probable  fault? 

(d)  What  Is  the  third  most  probable  fault? 

Decode  Test  Number  203310  of  Test  Tape  Request  Number  839902  and 
answer  the  following  questions. 

28)  Describe  the  function  of  each  part  of  the  program. 

29)  What  is  the  part  number  of  the  Self-Test  Adapter? 

30)  List  the  following  relays  by  reference  designation: 

(a)  Test  Point  Selection 

(b)  Test  Point  Distribution 

(c)  Miscellaneous  Relay/ s 

31)  What  is  the  number  of  events  (pulses)  during  the  time  period? 

32)  What  is  the  frequency  (PRF)  of  the  measured  signal? 

PART  II. 

33)  How  does  the  converter  store  information  into  the  computers? 

A.  Digital-Parallel  C.  Parallel-Digital 

B.  Serial  D.  Digital-Analog 

34)  The  converter  processes  what  type  of  data  from  the  computers  to 
provide  analog  outputs? 

A.  Analog-Digital  C.  Serial  Digital  Data 

B.  Discrete  Data  D.  Parallel  Digital  Data 

35)  What  type  of  Inputs/outputs  function  as  status  and  control  signals? 

A.  Synchro  C.  Discrete 

B.  DC  D.  Serial 
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36) 


37) 


38) 


39) 


40) 


41) 


42) 


43) 


44) 


*5) 


The  data  processed  by  the  converter  is  in  the  form  of  synchro,  AC, 
DC,  and  resolver  signals.  They  are  of  vhat  type? 


A.  Analog  Data 

B.  Serial  Data 


C.  Digital  Data 

D.  Discrete  Data 


What  is  the  advantage  of  parallel  digital  over  serial  digital  data? 


A.  More  components 

B.  More  wire 


C.  Time 

D.  Amount  of  data 


The  amount  of  times  we  do  a  given  signal  is  known  as  the: 


A.  Command  Pulse 

B.  Rate  Croup 


C.  Data  Reauest 

D.  Acknowledge 


Which  area  of  the  converter  controls  the  WDC? 


Area  I 

Area  I  and  III 


C.  Area  III 

D.  Area  II 


What  starts  the  rate  group? 


A.  Command  Pulse 

B.  Data  Request 


C.  Acknowledge 

D.  Parity 


The  arithmatic  section  of  a  computer  is: 


A.  Central  Processing  Unit 

B.  Core 


C.  Input/Output 

D.  None  of  the  above 


The  purpose  of  the  05  tape  is  to  check: 


A.  Inputs 

B.  Bite 


C.  Outputs 

D.  Converter  Set  Overloads 


How  many  outer  limit  checks  are  you  allowed  to  fail  on  any  test? 


A.  Two 

B.  Three 


C.  One 

D.  None 


A1A2  (Power  Supply  Controller)  controls  the  power  out  of  which 

drawer? 


A.  Variable  Power  Control  (A1A5) 

B.  Power  Supply  (A2A9) 


C.  Micrologic  Power  Supplv  (A1A9) 

D.  Power  Supply  (A3PS1) 


The  logic  Power  Supply  (A1AS)  will  cause  which  display  on  DATAC  when 
the  112VDC  PWR  SUP  and  the  ±28VDC  PWR  SUP  is  tripped  off? 


A.  No  go  low 

B,  Combination  of  TRU  inhibit  and 
Switch  Fail 


C.  No  go  high  and  Cenpac  control 

D.  Switch  Fail 


46)  Switching  Concrol  Unit  (A3A6)  controls  the  setting  of  routing  relays 

inside  the  following  drawers: 

A.  FCS  Adapter  (A4A1)  C.  Ratio  Transformer  (A4A6) 

B.  Transformer  Converter  (A4A2)  D.  All  the  above 

47)  If  an  adapter's  schematic  cannot  be  found  in  the  station  T.O. ,  then 

it  can  be  found  in  which  T.O.? 

A.  T.O.  35-1-181  F-lll  Miscella-  C. 

neous  Aerospace  Ground  Equip¬ 
ment,  WRALC  Multiple  Part  D, 

Numbers 

B.  T.O.  35-1-181-2  F-lll  Miscella¬ 
neous  Aerospace  Ground  Equipment 
SMALC  Multiple  Part  Numbers 

48)  The  VD^  in  Stimulus  Relay  Can  controls  the: 

A.  Setting  of  a  relay  supplying  C.  Information  Input  lines 

+28  vdc  to  the  can  D.  Steering  Input  lines 

B.  Setting  of  relay  supplying 
±12  vdc  to  the  can 

49)  A  VD4,  which  is  continuously  shutting  off,  can  be  fixed  by: 

A.  It  cannot  be  fixed  C.  Adjusting  the  Variable  Resistor 

B.  By  adjusting  the  Variable  clockwise 

Reglstor  counterclockwise  D.  Removing  it  from  the  Relay  can 

and  allowing  it  to  cool  down 

50)  A  Flip  Flop  needs  what  signals  to  set: 

A.  A  low  on  1,  high  on  0,  and  a  clock  pulse 

B.  A  high  on  1,  high  on  0,  and  no  clock  pulse 

C.  A  high  on  1,  low  on  0,  and  a  clock  pulse 

D.  None  of  the  above 

51)  Power  Supply  Sense  Lines: 

A.  Regulate  the  Power  Supply  at 
a  point  outside  the  Power 
Supply 

B.  Check  for  a  current  overload 
at  the  LRU 

52)  The  Emergency  Off  button  Is  located: 

A.  On  Cenpac  C.  On  the  Stimulus  Controller 

B.  On  DATAC  D.  On  the  Test  Point  Controller 


C.  Regulate  the  Power  Supply  at 
some  point  inside  the  Power 
Supply 

D.  None  of  the  above 


T.O.  33D7-38-1-112  F-111F 
Shop  Syscem 
A  or  B 


53)  The  Programming  131023*  34353651Info.  refers  Co: 

A.  Resecting  of  Stimulus  Relavs  C.  Setting  of  Relavs  034,  035, 

034,  035,  036  036 

B.  Setting  of  Relays  034  D.  Setting  of  Relavs  102,  343, 

536 

54)  A4A6  provides  Inputs  (during  maintenance  testing)  to: 

A.  DATAC  C.  A4A2 

B.  A4a7  D.  Cenpac 

55)  A  F02A250V4A  fuse  is: 

A.  A  slow  Blow  250  Volt  4  Amp  C.  Both  of  the  above 

B.  A  fast  Blow  250  Volt  4  Amp  D.  None  of  the  above 

56)  Is  it  permissible  to  replace  a  fast  Blow  fuse  with  a  slow  Blow? 

A.  Yes  C.  Sometimes 

B.  No  D.  Depends  on  the  amperage 

57)  Is  it  permissible  to  replace  a  fuse  of  a  given  amperage  with  one 
of  a  higher  amperage? 

A.  Sometimes  C.  No 

B.  Yes  D.  Depends  on  the  voltage 

58)  The  400  CPS  on-off  switch  on  A3A31  is  used  to  control: 

A,  Signals  applied  to  the  LRU  C.  Neither 

B.  Signals  applied  within  the  D.  Power  as  the  STA.  power  off 

station 

59)  The  DC  volts  meter  on  A1A2  measures  the  output  of  which  power  suppl 

A.  A2A5  Digital  Interface  Unit  C.  A2A3  Serial  Digital  Adapter 

B.  Parallel  Digital  Adapter  A2A4  D.  None  of  the  above 

60)  Power  Stimulus  Relays  can  be  tested  at  the  patch  panel: 

A.  In  the  usual  manner  C,  They  cannot 

B.  By  programming  a  Test  Point  D.  By  jump  2  sots  of  contacts 

Relay  together 

61)  Stimulus  Relays  can  be  tested  at  the  patch  panel  by: 

A,  Programming  the  Relay  and  ohms  checking  between  the  normally 
open  contacts  and  the  wiper 
B>  They  cannot  be  chocked 

C.  Programming  the  Relay  and  ohms  checking  between  the  normally 
closed  contacts  and  the  wiper 

D.  A  or  C 
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62)  Test  Poinc  Relays  can  be  checked  at  the  patch  panel  by: 

A.  Programming  the  Relay  and  ohms  cheeking  between  the  contacts  and 
A  and  C  of  the  DVM  Input  Plug 

B.  Programming  the  Relay  and  ohms  checking  between  the  two  patch 
panel  pins 

C.  Programming  the  Relay  and  ohms  checking  between  either  pin  and 
chassie  ground 

D.  None  of  the  above 

63)  A  coil  will  usually  ohms  check: 

A.  As  an  open  C.  A  very  high  number  of  ohms 

B.  As  one  or  two  ohms  D.  It  cannot  be  ohms  checked 

64)  A  Rectifier  is  used  to: 

A.  Change  AC  to  DC  C.  Balance  the  line 

B.  Change  DC  to  AC  0.  None  of  the  above 

65)  A  circuit  breaker  that  will  not  reset  usually  Indicates: 

A.  A  short  in  the  circuit  C.  A  shorted  filter  capacitor 

B.  A  broken  circuit  breaker  D.  All  or  any  of  the  above 

66)  If  you  are  not  sure  about  a  problem,  you  should: 

A.  took  at  the  T.O.  and  fake  it  C.  Push  start  on  DATAC 

B.  Ask  someone  D.  Co  for  a  coffee  break 

67)  A  pulse  period  is  measured  by: 

A.  An  o-scope  C.  DATAC 

B.  Counter  Timer  D.  A  or  B 

68)  The  Input  cable  on  A2A2  should  be  removed  when  trying  to  manually 
measure  a  voltage  with  the  DVM  because: 

A.  It  is  connected  to  the  back  of  the  DVM 

B.  It  doesn't  need  to  be  disconnected 

C.  It  will  induce  noise  on  the  line  - 

D.  All  of  the  above 

69)  The  front  panel  knobs  of  the  TRU's  can  be  enabled  by  the  DATAC  Mode 
Switch  in  which  position? 

A.  Normal  C.  Keyboard 

B.  Stop  D.  Manual 

70)  The  clock  input  for  A2A1,  A2A3,  A2A4  comes  from: 


A.  DATAC 

B.  Counter  Timer 


C.  Signal  generator 

D.  None  of  the  above 


BACKGROUND  DATA 


Section  A-5 


DRI  F-lll  TRAINING  SIMULATOR  EVALUATION  PROJECT 
AVIONICS  TECHNICIAN  SELF-RATING  FORM 


This  form  is  to  b«  used  to  assess  the  field  per¬ 
formance  of  the  technician  named  in  the  box  on  the 
right.  The  information  contained  cn  this  form  will 
in  no  way  be  used  to  affect  the  technician's  per¬ 
sonal  or  work  records,  cr  enhance  or  impede  his/her 
career.  This  form  becomes  the  sole  property  of  the 
Denver  He  search  Institute  and  KRL  under  the  terms 
of  Contract  :io.F33cl5-7a-C-0C18.  This  infestation 
is  being  collected  to  oonitor  the  field  performance 
of  those  technicians  whose  training  perforr.ar.ee  was 
monitored  at  Lowry  AT3  during  their  ATC  training 
assignment. 


1.  What  is  your  current  Test  Station  (7S)  assignment?  _ 

Approximately  how  many  weeks  have  you  been  assigned  to  that  TS?  _ 

2.  What  previous  TS  assignments  have  you  had  since  you  have  been  in  the  field? 


Approximately  how  many  weeks  were  you  at  each  of  those  assignments? 


3.  What  is  your  current  rank,  and  how  long,  in  weeks,  have  you  held  that  rank? 


•».  What  other  Air  Tcrce  bases  were  you  assigned  to  as  an  Avionics  Technician, 
prior  to  this  assignment ,  and  how  cany  weeks  were  you  at  each  previous 
assignment  (post-ATC  training  only)?  _ 


S.  During  the  6863  Test  Station  (Converter  flight  Control)  hlcek  of  your  ATC 

training  at  Lowry  ATS,  were  you  trained  or  tested  on  the  Training  Simulator? 
(Circle  the  correct  answer): 

a)  Yes,  trained  on  Simulator  e)  Tcs,  trained  and  tested  o:  Sim¬ 

ulator 

b)  Yes,  tested  cn  Simulator  d)  No,  no  contact  with  Simulator 

e)  Don't  knew 


TO  31  COMPiTTiD  3Y  DRI 


Technician  Last  Name: 

rirsor.il  Security  Humber: 

Current  AF2  Assignment: 

Technician  Tine  in  Field: 
3  mos.  6  res.  9  isos. 


On  the  following  pages,  we  vould  appreciate  your  help 
la  this  evaluation  project.  Flease  answer  the  questions ,  to 
the  best  of  your  ability,  usir.v  the  standardised  graduated 
scale.  The  questions  relate  to  your  current  working  situation 
and  your  ATC  training  at  Lowry  AF5.  Circle  the  "  x  "  on  the 
scale  which  most  accurately  reflects  your  situation  cr  opinion. 
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PREVIOUS  TRAINING  CURRENT  HELD  ASSICHfir.NT 


6.  How  many  hours  of  your  eight-hour  shift  0-1  2-3  4-5  6-7  8  or  more 

do  you  spend  operating  the  TS,  not  in-  x  ■ — — -*  — « - « 

eluding  observation  time? 


7.  In  operating  the  TS,  how  many  others 

none 

1 

2  3 

or  tore 

work  at  the  TS  with  you? 

X - i 

K - 

-X 

8.  How  much  individual  attention  do  you 

none 

some 

a  lot 

receive  on  the  TS  during  OJT? 

X - 1 

— K 

9.  How  comfortable  do  you  feel  operating 

not  at 

somewhat 

very 

the  TS  to  which  you  are  currently 

all 

ouch 

assigned? 

x - 

—at 

10.  How  much  attention  do  you  feel  should 

none 

some 

a  lot 

be  paid  to  troubleshooting  during  OJT? 

x - 

~X— 

-  X - X — 

— -X 

11.  How  much  attention  do  you  feel  should 

none 

some 

a  lot 

be  paid  to  theory  during  OJT? 

* - 

-H— 

— X 

12.  How  much  attention  do  you  feel  should 

none 

some 

a  lot 

be  paid  to  maintenance  during  OJT? 

» - 

— * 

13.  How  adequate  are  the  TOs  for  trouble¬ 

not  at 

somewhat 

very 

shooting  the  TS  and  LRUs? 

all 

u 

adequate 

14.  To  what  extent  does  OJT  address  your 

not  at 

somewhat 

very 

training  needs  usually? 

all 

such 

-JU- 

- X - X — 

— X 

15.  To  what  extent  does  your  current  field 

not  at 

somewhat 

very 

assignment  meet  your  personal/career 

all 

such 

objectives? 

X - 

X 

16.  How  relevant  was  your  ATC  training  to 

not  at 

somewhat 

very 

your  current  field  assignment? 

all 

such 

»  '■ 

-K— 

- X - X— 

- X 

17.  How  much  of  your  ATC  training  do  you 

utilize  in  your  current  field 

none 

some 

a  lot 

assignment? 

x - 

-X- 

- X  -  X- 

— X 

18.  How  adequate  was  your  ATC  trouble¬ 

not  at 

somewhat 

very 

shooting  training  for  preparing  you 

all 

such 

for  your  current  assignment? 

* 

19.  Kow  important  was  the  Converter  Plight 

not  at 

somewhat 

very 

Control  (6383)  block  of  your  ATC 

all 

isoortant 

training  to  your  current  assignment? 


20.  To  what  extent  does  the  training  you 
missed  at  ATC  (training  for  which  you 
received  deviations)  affect  your  ability 

to  operate  the  TS  in  your  current  assign-  not  at  somewhat  very 
sent?  all  such 

S - M - X - - - K 
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SIMULATOR  SPECIFIC  I  PREVIOUS  TRAINING 


21.  What  aspects  of  your  ATC  training  do  you  specifically  use  in  your 
current  field  assignment?  _ 


22.  What  aspects  of  your  ATC  training  do  you  use  very  little  in  your 
current  field  assignnent?  _ 


23.  What  capabilities  do  you  feel  a  three-level  avionics  technician  should 
have  when  he/she  begins  his/her  field  assignment?  _ 


24.  Do  you  feel  that  you  had  those  capabilities  as  a  result  of  your  ATC 

training  when  you  began  your  field  assignment?  YES _  NC _ 

If  not,  what  do  you  feel  you  were  inadequately  prepared  for? 


25.  What  would  you  add  to  the  overall  ATC  training  program  at  Lowrv  AT? 
to  better  prepare  avionics  technicians  for  their  field  assignments? 


The  remaining  questions  on  this  questior.aire 
are  directed  to  those  technicians  who  received  sore  training 
or  testing  on  the  Training  Simulator  at  Lcv.-rv  ATS.  If  veil 
circled  responses  £,  b,  or  £  cn  question  5  of  this  questicn- 
aire,  please  conplete  the  questions  to  follow.  If  vou  circled 
responses  d  or  e_,  you  have  completed  the  questions  ire  and 
should  return  this  form  to  your  supervisor. 


26.  To  what  extent  did  you  enjov  inter¬ 
acting  with  the  Training  Simulator 

not  at 
all 

some 

verv 

ruch 

during  ATC  training  at  Lowry  AFE? 

27.  To  what  extent  did  the  Training  Simul¬ 
ator  simulate  actual  TS  operating  con¬ 
ditions? 

not  at 

All 

sore 

very 

ruch 

28.  Do  you  feel  the  Training  Simulator  was 

probably  a  better  training  instrument 
than  the  actual  TS  for  ATC  training? 

29.  Do  you  think  simulated  training  instru¬ 

ments  should  he  used  more  frequently 
during  ATC  training? 

not  at 
all 

sore 

very 

much 

not  at 

all 

31  M 

sene 

verv 

much 
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30.  What  aspects  of  your  Training  Simulator  experience  did  you  find  part¬ 
icularly  enjoyable  or  beneficial?  _ 


31.  What  aspects  of  your  Training  Simulator  experience  did  you  find  part¬ 
icularly  bothersome  or  non-beneficial?  _ 


32.  What  would  you  change  about  the  Training  Simulator  to  make  it  a  better 
training  instrument?  _ 


You  have  completed  the  questionaire.  Please 
return  this  form  to  your  field  supervisor.  Thank  you  for 
your  assistance. 
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BACKGROUND  DATA 


Section  A-6 


DRI  F-lll  AVIONICS  TECHNICIAN  EVALUATION  PROJECT 


Supervisor  Rating  Form 


This  fcro  is  to  be  used  to  assess  the  field 
perfcroa~.ee  of  the  techniciar.  r.a-ed  in  the 
bo:-:  on  the  right.  The  ir.foro.ation  contained 
on  this  foro.  will  in  nc  way  he  used  to  affect 
the  technician's  personal  or  work  records,  or 
enhance  or  impede  his/her  career  status.  This 
forte  becor.es  the  scle  property  cf  IF!  and  HRL 
under  the  terms  of  Air  Force  contract  !.‘o. 
F3361S-78-C-C31E.  This  information  is  being 
collected  to  monitor  the  field  performance  of 
those  technicians  whose  training  performance 
was  monitored  at  howrv  AFS  during  their  ATC 
assignment. 


1.  Kane  of  person  rating  technician: (include  position,  e.g.  Sr.  Chief,  Shop 

Chief,  etc.) 

2.  Has  technician  left  the  Air  Force?  Yes _  Ko _ When?  _ 

3.  Test  station  technician  is  currently  operating  (e.s.,  6883,  6SS6,  6863,  etc.) 

<*.  Test  station  technician  has  operated  for  nose  of  field  duty:  _ 

5.  Other  test  station's  technician  has  operated  during  field  duty:  _ 

- - — - -  - * 

6.  Has  technician  been  reassigned  to  duty  other  than  TS  operation? 

Yes  ___  No _  If  yes,  what  other  duty  and  when? 


7.  Has  technician  received  a  promotion  during  current  period  of  rating: 
Yes  _  No _  If  yes,  from  what  level  to  what  level: 
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TEST  STATION  COMPETENCY  BASIC  TRAINING  COMPETENCY 


I;;  the  following  section,  please  rate  the  technician’s  task  performance  cr. 

•  scale  of  1  to  5,  where  1  signifies  very  poor  cr  unsatisfactory  performance, 
2  signifies  poor  performance ,  3  signifies  satisfactory  cr  fair  performance, 
u  signifies  good  perfomance,  and  5  signifies  very  good  or  excellent  perform¬ 
ance.  "A  indicates  this  task  is  not  applicable  in  rating  the  technician  at 
this  tine. 


S.  Practices  housekeeping  consistent  with 
tne  safer.-  cf  personnel  and  equipment: 

S.  Applies  safety  precautions  when  maintaining/ 
operating  test  station  equipment  (for  exam¬ 
ple,  removes  all  items  of  metal  such  as 
jewelry,  watches,  glasses,  etc.) 


10.  Uses  technical  manuals  as  a  source  of 
information  for  performing  maintenance 
and  inspections: 


11.  Uses  and  cares  for  common  test  equipment 
and  special  tocls  for  electronic  mainten¬ 
ance  applicable  to  tasks: 

12.  Performs  soldering  of  electronic  components: 

13.  Understands  purpose  and  function/operational 
concepts  of  test  stations  in  general: 

IN.  Understands  purpose  and  function/operational 
concepts  of  Cenpac: 

15.  Understands  numose  and  function  of  Computer/ 
Navigation  and  nieht  Controls /Con venrer  and 
night  Controls  t»st  stations  in  particular: 

16.  Understands  theory /signal  flow  of  particular 
LRUs  associated  with  test  stations  listed 
above: 

17.  Performs  confidence  test: 

18.  Operates  test  station  and  shop  standards  to 
perform  diagnostic  testing: 

19.  Operates  test  station  and  shop  standards  to 
troubleshoot  malfunctions : 


3  u 


3  4 


3  4 


3  4 


3  4 


3  4 


3  4 


S  NA 

5  KA 


S  NA 


5  NA 


5  NA 

5  KA 


5  NA 


5  NA 


20.  iterates  test  station  and  shop  standards  to 
perform  maintenance: 


3  4 


$  KA 
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ATTITUDE,  INITIATIVE  AND  SUPERVISION  TSC  (cont 


21.  Understands  general  LRU/SRU  information 

and  electrical  characteristics  for  LRUs/ 
SRUs  assigned  to  Computer/Kavieation  and 
Flight  Controls/Converter  Flight  Controls 
test  stations: 

22.  Understands  LRU/SRU  data  flow  and  interface 

with  test  stations  listed  above: 

23.  Prepares  special  requisitions,  issue  and 

turn-in  slips: 

24.  Coordinates  work  with  other  personnel: 

25.  Assigns  maintenance  and  repair  work: 

26.  Reviews  and  prepares  maintenance  and 

inspection  reports  and  charts: 

27.  Resolves  technical  problems  assigned  to 

him/her: 

28.  Works  well  without  significant  supervision: 

29.  Shows  initiative  in  troubleshooting  tasks: 

30.  Shows  gocd  work  attitude  toward  assigned 

tasks: 

31.  How  would  you  rate  this  technician  com¬ 

pared  to  other  technicians  at  the  same 
(3,5,7  or  9)  level  of  status? 


Very 

Poor 


fair 


Very 

Good 


NA 

NA 

NA 

NA 

NA 

NA 

NA 

NA 

NA 

NA 

NA 
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NOTES  ON  COST  ANALYSIS 


A)  Replacement  cost  of  space  estimate  was  taken  from  the  F-16  cost 
study  (Table  8).  The  estimate  of  $  16 . 27/square  foot  represents 
the  amount  the  building  inventory  at  Lowry  AFB  would  be  reduced 
at  the  end  of  1977  if  a  training  facility  were  scrapped.* 

B)  The  supplemental  furnishings  estimate  for  laboratory  space  was 
based  on  the  approximate  cost  of  items  used  in  the  laboratories. 

C)  The  acquisition  cost  estimates  for  the  AET  were  guided  by  the 
estimates  made  for  the  F-16  trainer.  The  acquisition  management 
cost  for  the  36th  unit  of  production  was  8.2  percent  of  the  unit 
cost  (Table  5).  Our  estimate  of  7.5  percent  was  more  conservative 
because  the  technology  of  the  F— 111  is  not  as  sophisticated  as 
that  of  the  F-16,  even  though  the  F— 111  station  represents  the 
eleventh  unit  produced. 

D)  Sustaining  Investment  estimate  was  guided  by  the  estimates  made 
for  the  F-16  trainer  which,  in  turn,  were  derived  from  Air  Force 
Logistic  Command  experience.  The  Sustaining  Investment  required 
was  found  to  be  approximately  12.5  percent  of  the  unit  cost  each 
year  (Table  5).  Our  estimate  of  $194,500  per  year  was  10  percent 
of  the  unit  cost  of  the  test  station.  The  unit  cost  was  inflated 
79.5  percent  to  reflect  wholesale  commodity  price  changes  from 
1972  to  1978  (source:  Economic  Indicators,  U.S.  Government 
Printing  Office,  Washington,  D.C.,  1979).  No  estimate  was  made 
for  logistics  costs  associated  with  the  spares  inventory. 


*The  F-16  study  refers  to  a  draft  technical  report, 
"Life  Cycle  Cost  Estimation  of  Simulated  vs.  Actual  Equipment 
Maintenance  Training  for  the  F-16  Avionics  Intermediate  Shop." 

This  report  was  prepared  by  the  AFHRL  Advanced  Systems  Division, 
Wright-Patterson  AFB,  Ohio  (T.  Eggenvier  et  al.,  March  1979). 

The  report  (unpaged)  was  a  substantial  resource  for  cost  data 
appropriate  to  training  at  Lowry  AFB,  and  the  equipment  cost 
model  employed  provided  substantial  insight  for  estimates  related 
to  acquisition  management  costs  and  the  logistical  costs  including 
Sustaining  Investment  for  weapons  subsystems. 
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E)  The  personnel  cost  of  maintenance  on  the  6883  AET  was  based  on 
records  of  actual  hours  spent  on  repair  from  1/1/79  to  7/30/79. 
(BLIS  Report,  Logistics  Branch,  Lowry  AFB) .  The  time  spent 
during  this  period  was  assumed  to  be  proportional  to  the  yearly 
maintenance  requirement.  The  grade  level  for  costing  purposes 
was  estimated  to  be  E-5. 

F)  Since  no  student  materials  are  required  for  6883  laboratory  work, 
this  cost  was  assumed  to  be  zero. 

G)  Updating  laboratory  exercises  was  assumed  to  require  one-sixth  of 
an  instructor  year.  Updating  represents  such  activities  as 
installing  "faults"  in  the  AET  to  facilitate  "manual"  trouble¬ 
shooting  exercises. 

H)  Instructor  contact  time  for  laboratory  training  was  estimated  at 
540  hours  per  year  (180  students  for  three  days,  eight  hours). 

The  estimate  was  increased  to  720  hours  per  year  to  better  account 
for  the  fact  that  one  full-time  instructor  would  be  assigned  to 
teach  6883  theory  and  practice. 

I)  The  overhead  or  burden  associated  with  maintaining  one  instructor 
one-third  time  was  estimated  by  apportioning  the  salary  of  the 
course  manager  (GS-12),  the  course  supervisor  (E-7) ,  and  the 
instructor's  supervisor  (E-6)  to  management  and  administration  of 
24  instructors  for  one-third  year.  (Salary  schedule  source: 
AFR-173-10,  1978.) 

J)  Miscellaneous  student  support  costs  for  students  in  residence 
at  Lowry  AFB  were  estimated  at  $79.25  per  student  per  week;  540 
student  days  represent  2.017  years,  assuming  260  working  days  per 
year  (2.077  x  52  x  $79.25  =  $8,550). 

K)  Estimate  based  on  AFHRL  project  records:  1056.5  hours  (military) 
and  2357.5  hours  (civilian)  prior  to  installation  of  the  simulator; 
cost  factors  for  military  and  civilian  personnel  for  AFHRL  were 
assumed  to  be  $12/hour  and  $14/hour  over  the  18-month  period. 

L)  Estimate  based  on  AFHRL  project  records:  1482  hours  (military) 
and  2374  hours  (civilian)  for  support  and  installation.  Same 
cost  factors  as  used  for  (K) ,  above. 

M)  Sustaining  Investment  for  the  6883  simulator  was  based  on  the 
cost  of  maintenance  contracts  in  force  from  June  1978  until 
November  1979,  and  a  contract  being  negotiated  for  1980-1981. 
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RELATIVE  COST  AND  TRAINING  EFFECTIVENESS  OF 
TIIE  <>HH3  TIIREK-DIMENSIONAI.  SIMULATOR  AND  ACTUAL  EQUIPMENT 

I.  IV  A7.  Table  15  should  be  changed  (asterisks  (*)  denote  changes)  as  follows: 

TABLE  15 

Tlit*  Life  Cycle  Cost  (Comparison 


(Cost  (Categories 


Simulator 


Facilities 

$1 10.650 

$110,650 

Equipment 

1.594.330 

4.902.140 

Instructional  Materials* 

26.000 

27.890 

Personnel 

04.250* 

72.530 

Students 

357.770 

357.770 

Miscellaneous 

0 

0 

Total 

$2,183,000 

$5,470,980 

NPV  (1078) 

$1,501,000 

$3,895,680 

Vsiiidciil-liour 

$348  =  $23/sludenl-hr.* 

$902  =  $60/studeui-hr.* 

2.  Page  70.  In  paragraph  I .  ($47  4/liour)  sliotthl  lie  changed  to  ($29/h«ur). 

3.  Page  110.  b  oot  mile  H  should  be  replaced  with  the  following: 

Instructor  contact  time  for  practical  training  on  the  6883  lest  station  was  esliinalrd  at  510  hours  per 
year  (based  on  approximately  li  hours  a  day  of  actual  student-instructor  classroom  interaction:  3 
day  s  of  instruction  |ht  class:  30  classes  per  year).  The  estimate  was  increased  to  720  hours  per  year 
to  more  accurately  account  for  the  fact  that  a  full-time  instructor  (8  hours  |>cr  day)  would  actually  In* 
assigned  to  leach  the  (>883  block  of  instruction. 

I.  Page  1 10.  In  line  3  of  footnote  J.  2.017  years  should  be  changed  to  2.077  years. 

E.  L.  ELLIOTT 

GliiiT.  Technical  Editing  Office 


